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Abstract 

We study the asymptotic law of a network of interacting neurons when the 
number of neurons becomes infinite. Given a completely connected network 
of firing rate neurons in which the synaptic weights are Gaussian correlated 
random variables, we describe the asymptotic law of the network when the 
number of neurons goes to infinity. We introduce the process-level empirical 
measure of the trajectories of the solutions to the equations of the finite net- 
work of neurons and the averaged law (with respect to the synaptic weights) 
of the trajectories of the solutions to the equations of the network of neurons. 
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The main result of this article is that the image law through the empirical 
measure satisfies a large deviation principle with a good rate function which 
is shown to have a unique global minimum. Our analysis of the rate function 
allows us also to characterize the limit measure as the image of a station- 
ary Gaussian measure defined on a transformed set of trajectories. This is 
potentially very useful for applications in neuroscience since the Gaussian 
measure can be completely characterized by its mean and spectral density. 
It also facilitates the assessment of the probability of finite-size effects. 

1 Introduction 

The goal of this paper is to study the asymptotic behaviour and large de- 
viations of a network of interacting neurons when the number of neurons 
becomes infinite. Our network may be thought of as a network of weakly- 
interacting diffusions: thus before we begin we briefly overview other asymp- 
totic analyses of such systems. In particular, a lot of work has been done on 
spin glass dynamics, including Ben Arous and Guionnet on the mathemati- 
cal side [291 El IH [30] and Sompolinsky and his co-workers on the theoretical 
physics side [391 HOI ECU EE2]- Furthermore the large deviations of weakly in- 
teracting diffusions has been extensively studied by Dawson, Gartner and 
co-workers [To"! \T7\ [To] . More references to previous work on this particular 
subject can be found in these references. 

Because the dynamics of spin glasses is not too far from that of networks 
of interacting neurons, Sompolinsky also succesfully explored this particular 
topic [38] for fully connected networks of rate neurons, i.e. neurons repre- 
sented by the time variation of their firing rates (the number of spikes they 
emit per unit of time), as opposed to spiking neurons, i.e. neurons represented 
by the time variation of their membrane potential (including the individual 
spikes). For an introduction to these notions, the interested reader is referred 
to such textbooks as [261 Ell 121] • In his study of the continuous time dynam- 
ics of networks of rate neurons, Sompolinsky and his colleagues assumed, as 
in the work on spin glasses, that the coupling coefficients, called the synaptic 
weights in neuroscience, were random variables i.i.d. with zero mean Gaus- 
sian laws. The main result obtained by Ben Arous and Guionnet for spin 
glass networks using a large deviations approach (resp. by Sompolinsky and 
his colleagues for networks of rate neurons using the local chaos hypothesis) 
under the previous hypotheses is that the averaged law of Langevin spin glass 
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(resp. rate neurons) dynamics is chaotic in the sense that the averaged law of 
a finite number of spins (resp. of neurons) converges to a product measure. 

The next theoretical efforts in the direction of understanding the av- 
eraged law of rate neurons are those of Cessac, Moynot and Samuelides 
[Ml [351 HQl [37]. From the technical viewpoint, the study of the collective 
dynamics is done in discrete time, assuming no leak (this term is explained 
below) in the individual dynamics of each of the rate neurons. Moynot and 
Samuelides obtained a large deviation principle and were able to describe in 
detail the limit averaged law that had been obtained by Cessac using the local 
chaos hypothesis and to prove rigourously the propagation of chaos property. 
Moynot extended these results to the more general case where the neurons 
can belong to two populations, the synaptic weights are non-Gaussian (with 
some restrictions) but still i.i.d., and the network is not fully connected (with 
some restrictions) |34j . 

One of the next challenges is to incorporate in the network model the 
fact that the synaptic weights are not independent and in effect often highly 
correlated. One of the reasons for this is the plasticity processes at work 
at the levels of the synaptic connections between neurons; see for example 
[32] for a biological viewpoint, and [HI [261 121] for a more computational and 
mathematical account of these phenomena. 

The problem we solve in this paper is the following. Given a completely 
connected network of firing rate neurons in which the synaptic weights are 
Gaussian correlated random variables, we describe the asymptotic law of the 
network when the number of neurons goes to infinity. Like in [341 [35] we 
study a discrete time dynamics but unlike these authors we cope with more 
complex intrinsic dynamics of the neurons, in particular we allow for a leak 
(to be explained in more detail below). The structure of our proof is broadly 
similar to these authors; we have generalised their results. Indeed one may 
directly obtain the LDP in [M] by applying a contraction principle to the 
LDP to be proved below. 

To be complete, let us mention the fact that this problem has already 
partially been explored in Physics by Sompolinsky and Zippelius [3H1 SO] and 
in Mathematics by Alice Guionnet [3U] who analysed symmetric spin glass 
dynamics, i.e. the case where the matrix of the coupling coefficients (the 
synaptic weights in our case) is symmetric. This is a very special case of cor- 
relation. The work in [13] is also an important step forward in the direction 
of understanding the spin glass dynamics when more general correlations are 
present. 
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Let us also mention very briefly another class of approaches toward the 
description of very large populations of neurons where the individual spikes 
generated by the neurons are considered. The model for individual neurons 
is usually of the class of Integrate and Fire (IF) neurons [33] and the un- 
derlying mathematical tools are those of the theory of point-processes [H] . 
Important results have been obtained in this framework by Gerstner and his 
collaborators, e.g. [23 125] in the case of deterministic synaptic weights. Re- 
lated to this approach but from a more mathematical viewpoint, important 
results on the solutions of the mean-field equations have been obtained in [8]. 
In the case of spiking neurons but with a continuous dynamics (unlike that of 
IF neurons), the first author and collaborators have recently obtained some 
limit equations that describe the asymptotic dynamics of fully connected 
networks of neurons [I] with independent synaptic weights. 

Because of the correlation of the synaptic weights, the natural space to 
work in is the infinite dimensional space of the trajectories, noted T z , of a 
countably-infinite set of neurons and the set of stationary probability mea- 
sures defined on this set, noted -M^ S (T Z ) . 

We introduce the process-level empirical measure, noted fi N , of the N 
trajectories of the solutions to the equations of the network of N neurons 
and the averaged (with respect to the synaptic weights) law Q N of the N 
trajectories of the solutions to the equations of the network of N neurons. 
The main result of this article (theorem |2]) is that the image law II of 
Q N through fi N satisfies a large deviation principle (LDP) with a good rate 
function H which is shown to have a unique global minimum, \x e . Thus, 
with respect to the measure Tl N on M J [ s iJ" E ), if the set X contains the 
measure 5^ e , then U N (X) — > 1 as N — > oo, whereas if 5 lle is not in the 
closure of X, U N (X) — > as N — > oo exponentially fast and the constant 
in the exponential rate is determined by the rate function. Our analysis of 
the rate function allows us also to characterize the limit measure /i e as the 
image of a stationary Gaussian measure /i e defined on a transformed set of 

trajectories S % . This is potentially very useful for applications since /i e can 
be completely characterized by its mean and spectral density. Furthermore 
the rate function allows us to quantify the probability of finite-size effects. 

The paper is organized as follows. In section [2] we describe the equa- 
tions of our network of neurons, the type of correlation between the synaptic 
weights, define the proper state spaces and introduce the different proba- 
bility measures that are necessary for establishing our results, in particular 
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the level-3 empirical measure, fi , II and the image R through /t of the 
law of the uncoupled neurons. We state the principle result of this paper in 
Theorem [2j In section [3] we motivate our approach by showing that when 
computing the Radon-Nikodym derivative of Q N with respect to the law 
of the uncoupled neurons, one is led to consider certain Gaussian processes 
which are directly related to the synaptic weights and can be described with 
the help of the empirical measure p, . 

In section [4] we extend the definition of the previous Gaussian processes 
to be valid for any stationary measure, not only the empirical one. This 
allows us to compute the Radon-Nikodym derivative of IT^ with respect to 
R N for any measure in M.~^ S {T 7L ). Using these results, section [5] is dedicated 
to the proof of the existence of a strong LDP for the measure LT^. In section 
[6] we show that the good rate function obtained in the previous section has a 
unique global minimum and we characterize it as the image of a stationary 
Gaussian measure. We conclude with section [7] by stating some important 
consequences and sketching a number of possible generalisations of our work 
as well as discussing some further connections with other approaches. 

2 The neural network model 

We consider a fully connected network of N rate neurons. For simplicity but 
without loss of generality, we assume N odd3 and write N = 2n + l, n > 0. In 
the course of this paper, we will asymptote N to oo, so that unless otherwise 
stated the parameters are taken to be independent of N. The state of the 
neurons is described either by the rate variables {X%), j = —n, ■ ■ ■ ,n, t = 
0, ■ ■ • , T or the potential variables (£//), j = —n, ■ ■ ■ , n, t — 0, • • • , T. These 
variables are related as follows 

Xi = f{Ui) j = -n,...,n t = 0,...,T-l, 

where / : R — >]0, 1[ is a monotonic bijection. We could for example employ 
f(x) = (1 + tanh(gx))/2, where the parameter g can be used to control the 
slope of the "sigmoid" / at the origin x = 0. 

We consider the case where the time variable t takes the T + 1 discrete 
integer values 0,1,..., T because it simplifies the problem. We leave for 
future work the case of the continuous time variable. 

1 Whcn N is even the formulae are slightly more complicated but all the results we 
prove below in the case N odd are still valid. 
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2.1 The equations 

The equation describing the time variation of the membrane potential CP of 
the jth neuron writes 

n 

Ui = yUi_ 1 + J2j ji f(Ut_ 1 ) + e j + Bi_ 1 , j = -n,...,n t — 1, . . . ,T. (1) 

i= — n 

This equation involves the parameters 7, J^, 8j, and B 3 t , i, j = —n, . . . ,n, 
t = 0,...,T-l. 

7 is a positive real between and 1 that determines the time scale of the 
intrinsic dynamics, i.e. without interactions, of the neurons. If 7 = the 
dynamics is said to have no leak. 

The JijS are the synaptic weights. Jij represents the strength with which 
the 'presynaptic' neuron j influences the 'postsynaptic' neuron i. They are 
random variables whose laws are described below. 

The 6jS are the thresholds: they change the value of the potential of the 
neuron j at which the sigmoid / takes the value 1/2. Like the JjjS they are 
random variables that we assume to be i.i.d. as J\fi(9, 9 2 ), and independent 
of the Jij% 

Finally the Bjs represent random fluctuations of the membrane potential 
of neuron j. They are independent random processes with the same law. 
We assume that at each time instant t, the B J s are i.i.d. random variables 
distributed as A/"i(0, a 2 ). They are also independent of the synaptic weights 
and the thresholds. 

The equation corresponding to ([T|) for the rates writes 

X{ = f [ir'ixU) + J ji X li + % + B li ) > ( 2 ) 

\ i=— n / 

where j = —n, ■ ■ ■ , n t — 1, ■ • • , T.. Note that the values of the rates are 
in the open interval ]0, 1[. 

2 We note M p {m, E) the law of the p-dimensional Gaussian variable with mean m and 
covariance matrix S. 
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2.2 The law of the synaptic weights 

The iV 2 synaptic weights are modelled as Gaussian random variables. We 
assume that they have the same mean which scales as 1/N: 

®[Jij] = -jT i, j = ~n,- • • ,n. (3) 

We next specify their covariance structure. The covariance is assumed to 
satisfy the following symmetry, 

COv(JijJkl) = COV^Ji+imj+nJk+TnJ+n) 

for all indexes i, j, k, I = —n, ■ ■ ■ , n and all integers m and n, the indexes 
being taken modulo N. We may interpret this property by imagining that 
the neurons are arranged on a ring with the following 'shift invariance'. If 
we fix two presynaptic neurons and shift the postsynaptic neurons, then the 
correlations are invariant. Similarly if we fix two postsynaptic neurons and 
shift the presynaptic neurons, the correlations are invariant. 

We stipulate the covariance through a function A : T? — > R, which satis- 
fies 

A(eifc, e 2 l) = A(k, I) £1, e 2 = ±1. (4) 
We assume furthermore that the covariances scale as 1/N. We write 

coviJijJki) = ((z - k) mod N, (j - I) mod N) . (5) 

Here, and throughout this paper, i mod is taken to lie between — n and 
n. It is important to note that the covariance function A and mean J are 
independent of N, so that these remain fixed when we asymptote N to infinity 
later on. We let A^ be the restriction of A to [— n,n} 2 , i.e. A N (i,j) = A(i,j) 
for — n <i,j< n. 

Being a covariance function (up to the scale factor 1/N) A N (i,j) must be 
a positive-definite function, i.e. it satisfies 

n 

A N {r- k,s-l)\ r8 \ k i > 0, 

r,s,k,l=— n 

for all reals X rs , A^, the indexing being taken modulo N. 



S 



The fact that is a positive-definite function imposes that its two di- 
mensional discrete Fourier transform (DFT), noted A^, is positive^], and also 
called its spectral density or power spectrum. In detail 

n 

A N {p,q) = A{k,l)e-^ pk+ql) p, q = -n, ■ ■ ■ ,n. (6) 

k, l=—n 

Conversely, the values of the covariances can be recovered from the Inverse 
DFT of the sequence (A N (p, q)) p ,q=- n ,~,n, i-e. 

1 n 

A (M = ^ E ^ N {P,q)e^ pk+ql) k,l = -n,-.. ,n. 

k, l=—n 

We must make further assumptions on A to ensure that the system is well- 
behaved as the number of neurons N asymptotes to infinity. We assume that 
the series (A(k,l))k,iez) is absolutely convergent, i.e. 

oo 

A sum = J2 |A(fc, /) | < OO. (7) 
k,l=— oo 

In practice one might expect there to exist a maximal correlation distance d 
such that A(k,l) = if \k\ + |/| > d (especially since in practice there is only 
a finite number of neurons). The existence of such a maximal correlation 
distance would be sufficient to guarantee the requirement (JTj), however we 
refrain from explicitly making this assumption as it is not necessary per se. 
It follows from (jZJ) that the series Ea*z=-oo A ( k i l)e~ i{ - kuJ1+l ^ is absolutely 
convergent and defines a continuous function, noted A(ui,u 2 ), on [— rr,n[ 2 
such that: 

A(k, I) = -J—r [ e *(fc"i+M A(wi, u 2 ) dui du 2 k,leZ. 

( 27r ) V[-7r,7rP 

The continuous function A is termed the spectral density of (A(k, l))k,ie%- 
It is clear from the definitions that, if the series (pn) and (qw) satisfy 
liniAr^oo 2tipn/N = lo\ and lmiAr^oo 2nq^ /N = u>2, then 

A N (p N ,q N ) A(u t ,u;2). 
3 This is a standard result in Fourier Analysis, see lemma [5] 
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Since is a positive-definite function (for all N), 

A(W1,W 2 )>0 V(Wi,W 2 ) G [-7T,7T[ 2 . 

We also assume that there exists A mm such that, for all A" 

A*(0, 0) > A min > 0. (8) 

2.3 The laws of the uncoupled and coupled processes 

The trajectories of the (X^)t=o-T defined by (j2j) are points in the T + 1- 

dimensional space ]0, 1[ [ °- T1 = T. The law of the solution of (J2J is a proba- 
bility measure on T N . We note Mf(T N ) the set of probability measures on 

j~N 

The trajectories of the (C/ J ')t=o,T defined by ([T]) are points in ]Rj ' T l = 5. 
The law of the solution of §B) is a probability measure on S N . This law is the 
image of the probability measure of that of the (X J ) t=0 ...T by the function /. 
We note Aif (S ) the set of probability measures on S N . 

2.3.1 The uncoupled processes and the initial conditions 

We specify the initial conditions for (J2J) as N i.i.d. random variables 
(Xo)j = - n ... n . Let fx i be the individual law on the interval ]0,1[ of X- 7 ; it 
follows that the joint law of the variables is /xf N on ]0, 1^. The initial 
conditions for (ITJ, noted (Uq) j=- n> ... >n , are also i.i.d. with law [x® N , where 

def 

fx = fXj o f is the image on R of the law /if defined on ]0, 1[ through 
the function /. Throughout this paper we employ the convention that if 
x e T then f(x) = (f(x ), . . . , /(x^)), and if x = (x~ n , . . . , x n ) G T w then 
/(#) = (f(x~ n ), . . . , f(x n )). We assume that /i is Gaussian under a change 
of variable, i.e. there exists ^o, a continuous bijection on R, such that 

^=^1(0,^)0*0, (9) 

We note P the law of the solution to one of the uncoupled equations (|2l) 
where we take ^- deterministic and equal to 9 and Jy = 0, i, j = —n, ■ ■ ■ ,n. 
P is the law of the solution to the following stochastic difference equation: 

X t = f (t/- 1 ^-!) + 9 + B t _x) , t = 1, • • • , T 
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with the law of the initial condition being Hence the image P_ of P 
through / is the law of the solution to the following stochastic difference 
equation 

U t = 7^-1 + £ + * = 1,--- ,T (10) 

the law of the initial condition being y, . This last process can be character- 
ized exactly, as follows. 

Let ^ : S — > S be the continuous bijection 

V(u) = t {v ,v ll ...,v T ), (11) 

where v = ^/ (u ) and for 1 < s < T, 

v s = ^ s (u) = u s - 7u s _i - 9 s = l,---,T. (12) 

We employ the convention that if u = {u~ n , . . . , u n ) e S N then = 
(^(n _n ), . . . , \l/(u n )). The following proposition is evident from equations 
©, (HOD and (PP. 

Proposition 1. The law P_ of the solution to ( flOl) writes 

P = Mt+i(0t+i, o- 2 ld T+1 ) o 

where 0<r+i t/ie T + 1- dimensional vector of coordinates equal to and 
Mt+i is the T + 1- dimensional identity matrix. 

2.3.2 Coupled processes 

If we reintroduce the coupling between the neurons, we note Q N (J,9) the 
conditional law of the (X$), j = —n, ■ ■ ■ , n, t — 0, ■ • ■ , T, solution to (J2]), 
for given (J, 0). It is an element of Mf(T N ). Similarly Q N (J,9) is the 

corresponding law of the (C//), j = —n, ■ ■ ■ , n, t — 0, • • • ,T, solution to (JTJ) , 
for given (J, 9). We let Q N = E J ' e [Q N (J, 6)] and Q N = E J ' e [Q N (J, 6)] be 
the laws averaged with respect to the weights and thresholds. 

2.3.3 Infinite number of neurons 

We note T % the set of doubly infinite elements of T. If x = (x J )j = _ 00j ... i00 
is in T z , we note x l , i G % its ith coordinate. We define the projection 
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ti n : T % ->■ (iV = 2n + 1) to be ir N (x) = (x~ n , . . . , x n ). The shift 
operator S : T z — > T z is defined by 

{Sx) l = x l+1 , teZ 

Given the element (x~ n , . . . , x n ) of T N we form the doubly infinite peri- 
odic sequence 

x{N) = (..., x n ~\ x n , x~ n , ...,x n , x~ n , x~ n+1 , . . .) 

which is an element of T z . We have (x(N)Y = x^ mod N \ where i mod N 
lies between — n and n. 

We equip T K with the projective topology, i.e. the topology generated 
by the following metric. For x, y G 7" , let 

d N (x,y)= sup |a^-J/2|. 

b'|<n,0<s<T 

This allows us to define the following metric over T s , whereby if x, y G T z , 
then 

oo 

d(x, y) = 2~ N d N {-K N x, 7Tjv?/). (13) 

AT=1 

The metrics djy and d generate, respectively, the Borelian sigma-algebras 
B{T N ) and i3(T z ). We note that T z is Polish (a complete, separable metric 
space) . 

A strictly stationary measure fi on T % with its Borelian sigma-algebra 
satisfies 

fi(S(B)) = (i(B) VBeB(T z ). 
Its ^-dimensional marginal also satisfies 

li N (S(B)) = fi N (B) V£? G B{T N ) 

where for each x G T N we have defined 

Sx = tt n (S(x(N))). 

We note Aif^T^) (respectively M.f s (T N )) the set of strictly station- 
ary probability measures on T % (respectively on T N ). Note that everything 

4 In what follows, the iV-dimensional marginal /j, N of a measure fi in A4± S (T Z ) is such 
that (jt N =(jo 7t^y^ . 
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above applies mutatis mutandis to S z (respectively S N ) and M.i s (S z ) (re- 
spectively Mf tS (S N )). 

We now introduce the following empirical measure. Given an iV-tuple 
(x~ n , . . . , x n ) in T N we associate with it the measure, noted fi N (x~ n , . . . , x n ), 
in Mf(T z ) defined by 

1 n 

^ :T N ^ M\{J % ) such that dfi N {x~\ ■ ■ ■ ,x n )(y) = ^ E S **<n)(v)- 

i=—n 

(14) 

We also sometimes consider pr to be a function on T z through the projection 

tt n : r z -> r". 

Note that fi N (x~ n , . . . , x n ) is a strictly stationary measure on T % . 
We introduce the following definition and notation. 

Definition 1. For each measure \i G Aif s (T N ) or A^^" S (T Z ) we define /x to 
be /x o /. 

We next equip .M^ S (T S ) with the topology of weak convergence, as fol- 
lows. This can be defined in many ways, but the following definition is 
the most convenient for our paper. For fi N ,v N G ■Mf s (T N ), we note the 
Wasserstein distance 

d N (ji N ,v N )=w£ j {E c (d N (x,y))}, (15) 

where J is the set of all measures in AifCT 21 ^) with iV-dimensional marginals 
fi N and v . For fi, v G M.f s (T z ), we define 

oo 

d(jJL,u) = 2^2 Kn d N ( f JL N ,U N ), (16) 
n=0 

where iV = 2n + 1. Here K n = max(A n ,2 _Ar ) and A n = ^fcL-oc I-M^j 71 )!- 
We note that this metric is well-defined because d^i^ , v ) < 1 and 

K ™ < 00 • ft can ^ e shown that .M^(T S ) is Polish. The topology 
corresponding to this metric generates a Borelian sigma-algebra which we 
denote by i3(.M^(T z )). The Borelian sigma-algebra on the set of stationary 
probability measures is denoted by B(A4f S (T Z )) . 

The construction of the topology of M.~t s (S N ) is analogous, except that 
in ( lT3"j) . for u,!)6 5^, we must replace dx{u,v) by 

d N (u,v)/(l + d N (u,v)), 
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the extra division being necessary because the finite-dimensional metric is 
unbounded. An analogous substitution must also be made in (TT5|) . 

Using the notation in definition [1] we note ft N the image of fi N through 
/. Hence, if u l = i = —n, ■ ■ ■ , n we have 

1 " 

i=—n 

where the shift operator is defined analogously on S z . Since the application 
\1/ defined in (fTTj) and (TT2"1) plays a central role in the sequel we introduce the 
following definition 

Definition 2. For each measure /i G A4f s (S N ) or .M^ s (iS s ) we define [i to 
be /jor 1 . 

Finally we introduce the image laws in terms of which the principal results 
of this paper are formulated. 

Definition 3. 



1. Let 11^ be the image law of Q N through the function p, N : T N — > 
Mt s (T z ) defined by (JT] 



2. We similarly define R N to be the image law of P® N under fi N . 

That is, Vfi e B{Mt >s {T % )), 

U N (B) = Q N (jl N eB) and R N (B) = P® N (fi N e B) . 

The principal result of this paper is in the next theorem. 

Theorem 2. LT^ is governed by a large deviation principle with a good rate 
function H (to be defined in definition^). That is, if F is a closed set in 
M+ S (T Z ) ; then 

lim" N' 1 log U N (F) < - inf H{pi). (17) 

Conversely, for all open sets O in ^At s (7^), 

lim iV^togn^fO) > - inf H(p). (18) 

By 'good rate function', we mean that H is not identically oo and the 
sub-level sets 

{/i e MtJ?*) : H{pL) < c}, 
where c > 0, are compact. 
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3 The Radon-Nikodym derivative of the av- 
eraged law of the coupled neurons with re- 
spect to the synaptic weights and the thresh- 
olds 

In the sections to follow we will obtain an LDP for the process with correla- 
tions (Q ) via the (simpler) process without correlations (P® N ). However in 
order for us to do this, we must first compute the Radon-Nikodym derivative 
of Q N with respect to P® N . It is easier to compute the Radon-Nikodym 
derivative of Q N with respect to P® N . We do this in the next proposition 
where, and we will use the same notation throughout the paper, the usual 
inner product of two vectors u and v of ]R T+1 is noted (u, v). 

Proposition 3. The Radon-Nikodym derivative of Q N with respect to P® N 
is given by the following expression. 



dQ 1 



dP 



E 



cxp 



1 / n 1 

- J](^),^)--ll^ll : 



\j=-n 




(19) 



the expectation being taken against the N (T + 1)- dimensional Gaussian pro- 
cesses (G l ), i = —n, • ■ ■ , n given by 

° l ° = ° - (20) 

Gl = £; = _ n %f«i)+^-0, t = l,--.,T, ^ 

and the function ^> being defined by (1111) and (1121) . 

Proof. The result can be obtained by an application of the Girsanov Theo- 
rem. We propose a detailed proof because it introduces important ideas that 
are used in the sequel. 

Let us define the iV random vectors (j = —n, ■ ■ ■ , n) of <S = 1R T+1 by 

Yi = V (U J ) 

Y> = Bl l + j = - n ,---,n t = l,---,T. 
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It can be seen that 

Y^N T+1 {eOZ T+1 ,a 2 \d T+1 ) j = -n,---,n, (21) 

where 0Z^ +1 is the vectoi0 of R T+1 whose first coordinate is equal to 
and the last T are equal to 1. Note that the variables are mutually 
independent. 

For fixed (J, 6), we let R Jfi : R 7V(T+1) ->■ R, N V +V > be the mapping u ->■ y, 

i.e. 

R Jt o(u- n ,---,u n ) = {y- n , ■■■,y n ) 

such that 

for j = —n, ■ ■ ■ ,n. Since the determinant D of the Jacobian of Rj^ is equal 
to rij=-n ^o( u o)' which is non zero by definition of \l/o, Rj,e is a bijection of 

R N(T+1) intQ itgelf 

Let <y9 e Cb(S N ) and let us compute 

5( J, (9) = e [^rr*. ■ ■ • , tP) | (J, e)] = ^e ■ ■ • , | (J, 0)] . 

Since (j = — n, ■ ■ ■ ,n) are independent of (J, using fl2T|) we write 

S(J,9)= [ <f(Rji(y- n ,---,y n ))^ 
Js N u 

(2^ 2 )-^ exP — i — ^ — — l (nn^l 



\j=-n t=0 



Through the inverse change of variables 

(y~ n , ■ ■ ■ , y n ) —> (u~ n , ■ ■ ■ , u n ) = Rjl(y~ n , ■ ■ ■ , y n ) we write 



S(J,9) = I <p(u- n ,--- ,u n )x 

n T 



2cr 2 

j=-n t=l 



5 A more natural notation would have been ZOt+i but we prefer the former in reference 
to the celebrated wizard. 
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Here, 



i=-« *=i 

E E N - 7*4-1 - W + (GD 2 - 2K - T^'-i - *)G?'] 



j=-n t=l 

where we have used the definition in ( 120|) . After noting proposition [TJ we 
find that 

Js N 

where 

^,W»)( u -« ... , u n ) = exp 



S(J,9) = ip(u- n ,--- ,u n )ij (J ^(u- n r -- lU n )P^(du- n ---du n ), (22) 



\i=-n 




Since (J22j) is true for all <p G C 6 (R Ar ( T+1 )) we conclude that 
dQ N (J,6 



\j=-n 



u~\ ■ ■ ■ , u n ) = exp - > (*(«'), G»> - -||GP 



— \j=-n 

where Q N (J,0) is the regular conditional probability of Q given (J, 6). By 
taking the expected value with respect to (J, 9) we obtain ( |T9|) . □ 

Note that the Radon- Nikodym derivative does not depend upon {^f (u^)}, 
j = —n, ■ ■ ■ ,n. We could have worked with T- dimensional processes & at 
the cost of making the last part of the paper heavier on notation. 

We now study the Gaussian system (G*)j=- n ,...,n,s=o,...,T in more detail. 
Throughout the rest of this section, we consider the u G S (in terms of which 
the system is defined) to be fixed, as is x G T, where x{ = f{u{). It will be 
seen that we may write the mean and covariance of the system as a function of 
the empirical measure fi N (x) (defined in (fT4l) ). This is of crucial importance 
because it will mean that the image laws of P® N and Q N under fi N , i.e. R N 
and U N , have Radon-Nikodym derivative given by the push-forward of fp 0N ■ 
For /i G M\{T % ), we define 

° t = ° (23) 
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Similarly, for fi N G Mf(T N ) let be the N(T + 1) x N(T + 1) block 
circulant matrix with ith block given by (for i = 



,N 



^ + El=- n A"(», m) j TN yUy™, V N (dy) 
K?;> 1 ={ for s, t = !,-■■ ,T (24) 

otherwise. 

Proposition 4. Fix x G T N and let u G S be such that x{ = f{u{). 
The covariance of the Gaussian system (G l s ), where i = —n, . . . ,n and s = 
0, . . . ,T writes ( x )) , where (fi N (x)) N is the N -dimensional marginal of 
fi N (x). For each i, the mean of G % is d 1 W. 



Proof. The mean of G\ is if t = 0, or otherwise is equal to 

j n T n f 

E pa = jj e = ^ E ^-i = ^ / ^ 7v (^)(y), 

jr'=-n J=-n ^ 

for t = 1, • • • , T. This is indeed independent of the index i. 

Let us now examine the covariance function K of these N Gaussian pro- 
cesses. It is an N(T+ 1) x N(T + 1) matrix which has a block structure, each 
block K lk , i, k = —n, ■ ■ ■ ,n, being the (T + 1) x (T + 1) covariance matrix 
of the two processes G l and G k . We have 

kT s = o, 

if s or t is equal to 0. We deal with the case where s and t differ from in 
the remaining of the proof, i.e 

n 

K* = cov{G\G k s ) = cov(J l3 J kl )xl_ 1 x l s „ 1 + 6\_ k ,s,t = l,--- ,T. (25) 

j,l=-n 

Because of our definition (ED of the covariance structure we have 



k 



•n=—n \ i=—n J 



Since K lk depends only on (i — k), it can be seen that K is a block circulant 
matrix, and we may write 

X ik = f if mod N 
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where we recall that j mod N lies between ±n. It follows from the symmetry 
of in 01]) that the matrix K % is symmetric, i.e. K\ s = K l st . It also follows 
from (j3J) that K~ % = K l . It may be inferred from (1251) that 

K\ s = 9% + J2 A "fr"0 I Vt-iVT-x tt N (x)) N (dy). 

Jr N 

m=—n ' 

□ 

We note that W) N is positive as (by definition) it is the covariance 
matrix of a Gaussian system. For fi N G A4i s {T N ), we denote the discrete 
Fourier transform of (K^'i), j = —n, . . . , n by (K^ > l ), i.e. for — n < I < n, 

n 

f>u N I tsu n i - 2nijl 

j=-n 

n 

K^'^N- 1 ^^'^. (26) 

l = — n 

We employ the convention of denoting the discrete Fourier transform of a 
sequence by a tilde throughout this paper. We state a basic result from the 
theory of block-circulant matrices, noting that the matrix indexing is from 
— n, . . . , n. 

Lemma 5. Let B be a symmetric block-circulant matrix with the (j, k) (T + 
1) x (T+ 1) block given by (£0'- fc ) modJV ) ; j, k = -n, ■•• ,n. Let B k = 
Y^=- n B l ex P {-^rr) f or 1^1 ^ n > and be the NxN Hermitian matrix 

with elements Wjjp = ^exp(^^), j, k = — n, - ■ ■ ,n. Then B may be 
'block'-diagonalised in the follow manner (where Cg> is the Kronecker Product 
and * the complex conjugate), 

B = (fW ® ld T+1 )diag (B~ n , B n \ {W {N) <g> Id T+ i)*. 

We observe also that A is an eigenvalue of B if and only if X is an eigenvalue 
of B k for some k. Finally the sequence (B^) (j = —n, . . . , n) is both real and 
even if and only if the sequence (B k ) (k = —n, . . . ,n) is both real and even. 

We now provide another form of equation f|T9|) by applying to it the follow- 
ing lemma from Gaussian calculus [3H ESj which we recall for completeness: 
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Lemma 6. Let Z be a Gaussian vector of W with mean c and covariance 
matrix K. If a € W and b e R is such that for all eigenvalues a of K the 
relation ab > — 1 holds, we have 



E 



l aZ - ^\\Z\\ 2 



1 



xexp (^ac - ^||c|| 2 + ^ \a - bc)K (Id p + bK)~ l (a - bc)^j 



v/det (Id p + bK) 

This leads us to the following proposition. 

Proposition 7. T/ie Radon-Nikodym derivative of Q N with respect to P® N 
is also given by the following expression, 

dQ N 1 



d ^ N y/det (W^cr+i) + £#<AW) 

^fe(i:(<^*M)-^ w ii a ) 

\ \i=— n 



i,j=-n 



Here A^ N ^ N ' 13 , i, j — —n, ■ ■ ■ ,n are defined to be the (T+ 1) x (T + 1) blocks 
of the N(T + 1) x iV(T + 1) matrix K^ N ' (a 2 ld N{T+1) + K^ N )-\ 

Proof. The eigenvalues of M) are positive because it is a covariance 
matrix. We thus obtain our result by the application of lemma [6] to equation 
(HHD with p = N(T + 1),Z= (G~ n , G~ n+ \ • • • , G n ), 

a = ^(^( u - n ), ^(u~ n+1 ),--- ,*(w n )), i = -n,- ■■ ,n, and 6= □ 

We define the subset -M^(T Z ) = U* 6 r" C -M^ S (T S ). The 

definition of .Mf^S 2 ) is analogous. 

The righthand side of (1271) is the product of two terms which we analyse 
in some detail in order to prepare the ground for the definition of a rate 
function. 

We first note that the first term can obviously be rewritten as described 
in the following lemma. 
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Lemma 8. The following relation holds: 

1 

'det (ld N(T+1) + ^(AW) 
where for /i e .Mf^CT 2 ), we define 



1 , 



The above expression has sense because the eigenvalues of Idjv(T+i) + 
a are bounded below by 1. We next express the second term as a 

function of the empirical measure fx N . This is done by elucidating the block 
structure of the matrix A& ( x " as revealed in the following lemma. 

Lemma 9. The matrix = K^ N ^ N (a 2 ld N{T+1) + K^W)- 1 is 

symmetric and block circulant. It is built from an even sequence of (T + 1) x 
(T + 1) symmetric matrixes ( x >) > l , i = —n, ■ ■ ■ ,n. Furthermore we have 

ex p (^(e((^U(- i ))-^i^ ) ii 2 ) + 

- J2 (^(u i )-c^ {x \A^ N{x))N ' lj ^(u j )-c^ N{x) )))\ = exp(NT 2 (fi N (x))), 



2 

h3=-n 



where for ji <G M.x's"^T % )> we define 



2(cf t , V(v )) - ||c1 2 fj, N (u)(dv). (29) 



Proof. It can be seen that the matrix A& ^ is block-circulant through the 
diagonalisation of (given in lemma E]) in the definition of A& ( x » . 

We index the blocks as A^ ( x » > % , % — —n, ■ ■ ■ ,n, where 

A (fi N (x)) N ,ij = A (fl N (x)) N ,(i-j) modJV 



21 



The diagonalisation in lemma allows us to write 



This means that ^ ,J = ^ ' J and the blocks are symmetric 
(since these properties apply to the blocks of ). In turn, this means 

that A^ N W N >-3 = A^ N W N 'i and *A(A N (*))^ = AV"W N >i. The result 
now follows from a substitution of the definitions. □ 

It is useful to put together proposition [7J lemma |8] and lemma [9] in the 
following proposition. 

Proposition 10. The Radon- Nikodym derivative of Q N with respect to P^ N 
writes 

dO N 

(u~ n , ■•.,«») = exp(NT((fi N (f(u- n ), • ■ ■ , f(u n ))) N )), 



d p®N 

where for /i G A^^ + (T Z ), r(/i 7V ) = T 1 (/i Ar ) + T 2 (fi N ) and the expressions for 
T\ and T 2 are given by lemmas\E and\Q 

Before we close this section we define a subset of M.~t s which appears 
naturally. 

Definition 4. We define the subset £ 2 of Mf^T^) by 

£ 2 = {fi£MUTZ)\E^(u°)\\ 2 ]<oc}. 

For this set of measures, we may define the stationary process (v k )kez in 
<S Z , where v h = ^/(u k ). This has a finite mean E=[t>°], noted v**, where we 
recall from definition [2] that /j = p It admits the following spectral 

density measure, noted such that 

E£[v ot v k ] = — e lkul v»(duj). (30) 

We similarly define 

sf ] = {/ig mut n ) < oo}, 

and note that if /j £ £ 2 then n N e i^^- 
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4 The image of the averaged law through the 
empirical measure 

In the previous section we saw that the Radon-Nikodym derivative of Q N 
with respect to P® N may be expressed as a function of the empirical measure, 
i.e. exp(NT((fi N (x)) N )). In this section we obtain an expression for the 
Radon-Nikodym derivative of the image laws II and R N under the empirical 
measure. We do this by extending T beyond the range of p, to an arbitrary 
measure m Ml s {T z ). We will see that f^G") is exp(NT(fj, N )), and that 
the extended function T(/j, N ) is lower semi-continuous. 

4.1 Gaussian processes 

We determine the Radon-Nikodym derivative at fj, N by writing T as a function 
of a Gaussian process G 11 which is, in turn, determined by fi N . We begin 
with finite N, before proceeding to the infinite-dimensional projective limit. 

4.1.1 Finite number of neurons 

Given /j, N in Aif s (T N ) we define the stationary N(T + l)-dimensional Gaus- 
sian process C . We will use G" to define T{fi N ). 

In analogy to fl23|) . the mean of ,l is equal to if t = 0, or otherwise 

cf = J yl-ifJ> N (dy), t = 1, • • ■ , T, i = -n, ■ ■ ■ , n. 

JT N 

We note that the above integral is independent of % due to the stationarity 
of fi N , which is why we have omitted the superscript i from . 

For each pt N e Ml s (T N ) let M^> fc , (k = -n, ■ ■ ■ ,n), be the (T + 1) x 
(T + 1) matrix defined by (for s,t G [1,T]), 

' k = [ vUvti^ \dy). (31) 
Jt n 

N i. 

If s = or t = 0, then Af£ ' = 0. 

The covariance matrix of G M is defined by the block circulant N(T + 
1) x N(T+1) matrix , which has blocks given (in analogy with equation 

<m) by 

n 

K£* = - 5 S ){1 -S t )+J2 A(z,m)M s f ' m , (32) 
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for i = — n, ■ ■ ■ ,n. The discrete Fourier transform (K^ 1 J ')of the sequence 
(K^ N ' 1 ) is given by <0§. 

We now state some properties of the matrices we have just defined. We 
will prove that is positive, which means that it is a well-defined covari- 
ance matrix. 

Lemma 11. For all k = —n, ■ ■ ■ ,n, 

t M v N ,k = M" N ~ k . (33) 

The blocks (K^ N ' k ) are symmetric and satisfy K^ N ~ k = K^ N ' k . Furthermore 
M^ N ' k (for all k = —n, ■ ■ ■ ,n), ,k (for all k = —n, ■ ■ ■ ,n) and are 
all positive. 

Proof. The identity ( 133]) follows directly from the stationarity of y, . The 
evenness of the series (K 11 '*), i = —n, . . . ,n, follows from (J3J) and (I3"2~j) . It 
follows from ( 13 3 p that 

n 

Kg'* = 9%(1 - 5 S )(1 -5 t )+"£ Hh m)M? s N >~ m . 

m=—n 

We find, in turn, by fll]), that 

n 

Kg* = 6%(1 - 8 S )(1 -8 t )+ A^m)M? s N > m = K£>\ 

m=—n 

which means that K^ N ,l is symmetric. 

Let M M be the block-circulant matrix with blocks given by M M ,k , k = 
—n, . . . , n. Let r : T N — > T N be the map such that for all \k\ < n, t(x)q = 
and for s G r{x) k = x k _ 1 . It may be observed from ( )3~Tj) that M M 

is the correlation matrix of fi o r _1 , which means that it is positive. It then 
follows from lemma that the matrixes 

71 

„~V,,JV ; \ — ^ , r .,N u 2-nikl 

k=—n 

are positive. We also observe from this lemma that 

n 

M^ fe = -^M^W. (34) 

l=—n 
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To prove that the matrix is positive, it suffices (by lemma E]) to prove 
that ,l is positive for all I = (— n, . . . , n). Using ( I34p . we write 



1 n / n n 

= ^oz T+1 *oz T+1 + i y: E E A ( fc ' 

p=—n \rn=—n k=—n 



2iri(kl — mp) \ ~ TV 

m)e ^r— 1 M" ' p . 



Using the symmetry A(k, —m) = A(k,m), this can be rewritten in terms of 
the spectral density A N of A, i.e. 



1 n 

k* > l = # 2 oz T+1 'oz T+1 + ^ E ^ n (i,pW" ' p . 

p= — n 

Hence, for W G S, we have 

'WK^^W = 9 2 (OZ T+1 , W) 2 + — ^ N ( l >P) (^WM^^Wj . (35) 



p=—n 



This is positive because the spectral density A N is positive and t WM^ ' P W 
is positive. □ 

4.1.2 Infinite number of neurons 

Given /x in A^^ S (T Z ) we define a stationary Gaussian process G M with values 
in S % . It will be seen that is the limit of G^ N (where (jl n is the N- 
dimensional marginal of /x) as A" — > oo, in the sense that the means and 
covariances converge. 

The mean is the same as the finite-dimensional case. That is, for all % the 
mean of G^ % is given by cf , where 

c? = J [ yUdfj,{y),t = !,-■■ ,T,ie%, and c£ = 0, (36) 

the above integral being independent of i due to the stationarity of /x. 

We now define the covariance of G M . The definition of M M,fc is analogous 
to the previous definition, i.e. 

Definition 5. Let M^' k , k E 1 be the (T + 1) x (T + 1) matrix defined by 
(for s,te[l,T\), 

M»? = [ yUvlMy). (37) 
If s = or t = 0, then M£ h = 0. 
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These matrixes satisfy *M M,fc = M M ' k because of the stationarity of 
/i. Furthermore, they feature a spectral representation, i.e. there exists 
a (T + 1) x (T + 1) matrix- valued measure M M = (M^) m=0 ,...,t with the 
following properties. Each M^ t is a complex measure on [—ir, ir[ of finite 
total variation and such that 

1 r 

,,k I ikui 



M M,fe = _ / e ^M»{du). (38) 

Furthermore, for all vectors W G R T+1 , t WM(du)W is a positive measure 
on [— 7T, 7r[. 

The covariance between elements G 11 ' 1 and G^' l+k is defined to be 

oo 

= 9 2 5 k OZ T+l 'OZ T+1 + ^ A(fe, l)M»> 1 . (39) 

!=— oo 

We note that the above summation converges for all k G % since the series 
(A(k, l))k,ie% is absolutely convergent and the elements of M^' 1 are bounded 
by ±1 for all I G It. We next prove that the sequence (K^ k ) k& % admits 
a spectral representation (which in turn implies that is a well-defined 
covariance operator). 

Proposition 12. The sequence (i^' fe )kes has spectral density given by 
K"(uj) = 9 2 OZ T+1 t OZ T+1 + -*- / A(u, i)M(di). 

That is, is positive and satisfies 

K n,k = — [ e ikuJ K^(u)du. 
Proof. First we prove that the matrix function 

oo 



iku) 



is well-defined on [— 7r,7r[ and is equal to the expression in the statement of 
the proposition. Afterwards, we will prove that is positive. 
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From (1391) we obtain that, for all s,t € [0, T], 

oo 

\K£ k \<Td%+ £ \Hk,l)l (40) 



l=— oo 



This shows that, because by ([7j) the series (A(fc, is absolutely conver- 

gent, K^{uj) is well-defined on [— tt, tt[. 
Using ( 13"8|) we write 

1 PIT / °° 00 \ 



m=— oo fc=— 00 



Using the symmetry A(/c, — m) = A(k,m) this can be rewritten in terms of 
the spectral density A of A 

K»(uj) = 6 2 OZ T+l *OZ T+1 + — / A(w, 7)M M (^7). 

We note that is positive, because for all vectors W of R T+1 , 

1 WK^(uj)W = e 2 {OZ T+u W) 2 + j A(w,7) (^M^cfy)^) , 

the spectral density A is positive and the measure t WM^(d , y)W is positive. 

□ 

The finite-dimensional system 'converges' to the infinite-dimensional sys- 
tem in the following sense. In what follows, we use the Frobenius norm on the 
(T+l)-dimensional matrices. We write (u) = Yl^-n^ ' k ex P( — ikoj). 
Note that for \j\ < n, (2irj/N) = The lemma below follows 

directly from the absolute convergence of Y2j k I -Mi, 

Lemma 13. Fix fi G M.\ S {J~ % \ For all e, there exists an N such that for 
all M > N and all j such that 2\j\ + 1 < M, || A^ Mj ' - \\ < e and for all 
cue[-n,n[, \\K» M (u) - K»(u) \\ < e. 

Lemma 14. The eigenvalues of K^ N ,l and K^(u) are upperbounded by 

PK = (t + 1) (e 2 + A sum ) , 

where A sum is defined in ([7]). 
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Proof. Let W G S. In the finite-dimensional case, we find from (I35p . (E]), 
and d7j) that 



1 n 

'WK^JW < 9 2 (T + l)\\W\\ 2 + A sum — \^WM»"*W). 

p=—n 

= 6 2 {T + 1)|| W\\ 2 + A sum 



The eigenvalues of '° are all positive (since it is a covariance matrix), 
which means that each eigenvalue is upperbounded by the trace, which in 
turn is upperbounded by T + 1. Through taking the limit iV — > oo we also 
obtain the upperbound for K^(uj). □ 

We let A^ N = K^ N {a 2 ld N(T+1) + K^)- 1 . This is well-defined because 
is diagonalizable (being symmetric and real) and has positive eigenval- 
ues. Furthermore, it follows from lemma [5] that this is even block circulant, 
with symmetric blocks ,k {k = —n, ■ ■ ■ ,n) and that 



AM = A» N > k e-^ = kM {a 2 ldT+1 + kM)-\ (4i) 

k= 



In the limit N — > oo we may define 

A^u) = K»(ou)(a 2 ld T+1 + K^iu))^ 1 

as the product of two functions defined on [— tt, 7r[ whose Fourier series are ab- 
solutely convergent. The Fourier series of (aHdr+i + K^iu)) -1 is absolutely 
convergent as a consequence of Wiener's theorem because the eigenvalues of 
(j 2 Id r+1 + K M (w) are strictly positive. Hence the Fourier series of A^(u), i.e. 
(y4 At,fc ) fc6Z , is absolutely convergent. We thus find that, for 1 G Z, 

A* 1 = — f A"(u)e il0J duj = lim A""' 1 (42) 

and 

oo 

A»{u) = A^e~ ilw . 

l=— oo 

Let A^ (oj) = J2k=-n A^ ,k exp(— ikuS) and note that for \j\ < n, 
A» n (2tij/N) = i^V 
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Lemma 15. The map B — > B^aHd^+i + B) 1 is Lipschitz over the set 
A = {K^ N (u),K^(u) : fi G Mf tS (T®),N > 0, u; G [-7r,7r[}. That is, there 
exists a positive constant Au p such that for all -81,-82 G A, 

||S 1 ( f 7 2 Id T+1 + B,)- 1 - B 2 (a 2 ld T+1 + B 2 )~ 1 \\ < Afipp! - B 2 \\. 

Proof. The eigenvalues A of the matrixes in A satisfy < A < px- Thus, 
both B and (cr 2 Idx+i + B)~ l are bounded in the operator norm (which is 
equal to the largest eigenvalue) for all B G A. They are thus bounded over 
every matrix norm (as the matrix norms are all equivalent). The first term 
is clearly Lipschitz, and the second term is also Lipschitz because 



(o 2 Id 



T+l 



B 1 



,-1 



(a 2 ld 



T+i i - B 2 ) 1 

2 T J , D \-l 



o 2 Id T+1 + -BO" 1 (B 2 - B,) (a 2 ld T+1 



B,r l . 



□ 



The following lemma is a consequence of lemmas [13] and [151 



KM - K' 



a v s M 



Lemma 16. Fix fi G .M^ S (T Z ). For all e, there exists an N such that for 
all M > N and all u G [-7r,7r[, \\A^ M (lo) - < e. 

The above-defined matrices have the following 'uniform convergence' prop- 
erties. 

Proposition 17. Fix v G .M^ s (T a ). For all e > 0, there exists an open 
neighbourhood V e {v) such that for all fi G V e (u), all s,t G [0, T] and all 

LO G [-7T,7T[, 

(43) 

(44) 
(45) 

(46) 
(47) 



AM 



< e. 



< e, 



K-<£i< e , 

and /or a// JV > 0, and /or a// k such that \k\ < n, 



K 



St 



1 St 



<e, 



and 



7v»,k 
A st 



■ft st 



< e. 



29 



Proof. The bounds are evident if s = or t = as the elements are all zero, 
hence we may assume that s and t are nonzero throughout this proof. Let \x 
be in M J [ yS {l" E ) and uj £ [— 7r, 7r[. We have 



ix,k 
st 



k=—oo 



Using f[39|) we have 



hence 



- K(u) 



A{k,l)(Mg-M$)e 

k,l=—oo 



ikui 



< 



oo „ 

J2 |A(M)I / IxUxli-ylivLil^id^dy), 



k,l=— oo 



where L = 2|/| + 1 and C 2L has marginals fi L and v L . Since \x° s _ l x l t _ l 
Vs-iVt-il < 2d L (iT L x,iiLy), we find (through (TIE]) ) that 



Thus for ( l4"3"j) to be satisfied, it suffices for us to stipulate that 14(f) is a ball 
of radius less than |e (with respect to the distance metric in (116j) ). Similar 
reasoning dictates that (pESj) is satisfied too. 

However in light of lemma [151 it is evident that we may take the radius 
of V e {v) to be sufficiently small that (14"3)) . f H6l) and fj47j) are satisfied. In fact 
(]44p is also satisfied, as it may be obtained by taking the limit as N — > oo of 
(|47p . Since c M is determined by the one- dimensional marginal of /i, it follows 
from the definition of the metric in ffTSj) that we may take the radius of V e (u ) 
to be sufficiently small that (1451) is satisfied too. □ 

A direct consequence of the above proposition is that c M , , K^ 1 , A^ N 
and are continuous with respect to /i. 



4.2 Definition of the functional T 

We have previously (in ([8]) and ff29l) ) defined a functional r := T 1 + T 2 on 
the image of fi N . We now extend these definitions to functionals r^I^ : 
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M.\ S {T N ) — > R. It will be seen that these functionals asymptote to a limit 
as iV — > oo, so that we may consider 1^ and T 2 to be defined on -M+ s (T a ) 
as well. 

Let n G Ml s (T z ), and let (p N ) n>i be the iV-dimensional marginals of 
H (for N = 2n + 1 odd). 

4.2.1 r x 

We define 

Ti(^) = ~ log (det (ld N(T+1) + ) . (48) 

Because of lemma [11] the spectrum of K* 1 is positive, that of Idjv(T+i) + 
is strictly positive and the above expression has a sense. Moreover, 



riOO < o. 

We now define = lini7v->oo Fi(fi N ). The following lemma indicates 

that this is well-defined. 



Lemma 18. When N goes to infinity the limit of (1481) is given by 
IMaO = ~^J (det (ld T+ i + ^»)) ^ 

for all fie Mf tS (T z ). 

Proof. Through lemma we have that 



(49) 



(50) 



where we recall that & (^) = tf**" 1 . Since, by lemma H3 (w) con- 
verges uniformly to K**(lij), it is evident that the above expression converges 
to the desired result. □ 

Proposition 19. Ti is bounded below and continuous on both Aif s (T N ) and 
MUT*). 
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Proof. Applying lemma E]in the case of Z = (C ' n — , • • • , C" - & 1 ' 
a = 0, b = o~~ 2 , we write 



,N „ ,,JV^ 



r 1 ( / i JV ) = -i 0g E 



exp 



2a 2 ^ 11 



c mJV | |2 



k=—n 



Using Jensen's inequality we have 



2Na 2 



E 



£ ||G"* fc 



N i, ,,N 



p." II 2 



k=—n 



2o 2 



E 



By definition of K^' , the righthand side is equal to — ^Trace^^' ). 
From ( )32|) . we find that 

Trace^^' ) = TO 2 + ^ A(0, m)Trace(M^' m ). 



It follows from the definition ( T3TT) that 

< |Trace(M^' m )| < T. 

We obtain 



Trace^^' ) < T le 2 + |A(0,m)| ) < T (# 2 + A sum ) 



Hence 
where 



iM/x") > -Pi, 

01 = 7^1 i 02 + A " m ) 



(51) 



It follows from lemma [TBI that —fli is a lower bound for T^/i) as well. 

The continuity (over both Aif s (T N ) and A4f s (T z )) follows from the 

expressions ( )48j) and ( )49l) . continuity of the applications ji N — > and 
/i — > (proposition [T7|) and the continuity of the determinant. □ 
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4.2.2 r 2 



We define, analogously to (1291) . 



2a 2 



1 




+ 



2 (c" ,v°) - \\d 



N 



2 



) 



fji N (dv), (52) 



with 1 1 c M 



N 



' = EL (4") 1 



and n defined in definition [2j This quantity is 



finite in the subset £% of J\4f s (T N ) defined in definition HI If ji N £% , then 
we set r 2 ( / u. Ar ) = oo. 
We define 



where fi N is the N- dimensional marginal of /i. If /i ^ £ 2 then fi N ^ £% 
and r 2 (/i) = 00. We assume throughout the rest of this section that /i G 
M-i s (7" z ) is in £ 2 . This means that the spectral measure (as given in 
(130]) ) exists. The following proposition indicates that T 2 (/x) is well-defined. 

Proposition 20. If the measure fi is in £2, i- e - if E=[||w°|| 2 ] < oo ; then 
r 2 (/x) is finite and writes 



The ":" symbol indicates the double contraction on the indexes. One also 
has 



r 2 (/i) = iim r 2 oo 





+2E% / >°>] - ||c"|| 2 ) . 
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Proof. We note firstly that &• = d 1 . Using (l3"Uj) . the stationarity of \i and 
the fact that Y,k=-n A ^' k = ^"(0), we have 



+ \ I (c»,v ) - t c»A» N (0)v o d f i(v) + -^ t ci* (idT+i-A^iO))^. (53) 



From the spectral representation of we find that 

r 2 (^) = ^ r^( w ):W 



+ 4^ 



a 2 " 



V(Id T+1 - i^(0))c^] + (idr+i - i^(0)) c". (54) 



Since (according to proposition [T6|) ^^(w) converges uniformly to A^^oj) as 
iV — > oo, it follows by dominated convergence that T 2 (fi N ) converges to the 
expression in the proposition. 

The second expression for ^(yu) follows analogously, although this time 
we make use of the fact that the partial sums of the Fourier Series of A^ 
converge uniformly to A^ (because the Fourier Series is absolutely conver- 
gent). □ 

We next obtain more information about the eigenvalues of the matrices 
A tlN ' k = A^ N (^) (where k = -n, . . . , n) and A^uj). 

Lemma 21. There exists a < 1, such that for all N, \i andu, the eigenvalues 
of A 11 ,k , A^(u) and A^ are less than or equal to a. 

Proof. By lemmadH the eigenvalues of K^{u) are positive and upperbounded 
by px- Since K^(u) and (cr 2 Idr+i + K^(uj) J are coaxial (because is 
real and symmetric and therefore diagonalisable) , we may take 

Pk 

a - 



o- 2 + p K ' 



This upperbound also holds for A^ ' k , and for the eigenvalues of A^ because 
of lemma □ 
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We wish to prove that r 2 (/i Ar ) is lower semicontinuous. A consequence 
of this will be that r 2 (/i 7V ) is measureable with respect to B(M.i tS (T N )). In 
order to do this, we must first prove that its integrand possesses a lower 
bound. We do this by diagonalising A^ 1 into its spectral representation. 

We use the fact that the measure /i N is stationary to rewrite fl52j) in a 
more symmetric fashion, 



r 2 (//0 = ±f ( 1 ± ± V' - c-V^ V +J 

J sN \ k=-nj=-n 



,N 



c» ) + 



±^(v*-cf,c?)+\\cf\\A!f{dv), ( 55 ) 

j=-n / 

where we recall that the neuron-indexing is taken modulo N. Define the N 
(T + 1) -dimensional vectors 



w 



k = v k_ c ^ k= _ 1 



We use (jUj) to replace ' by its Fourier representation, so that the 
quadratic component of the above expression becomes 



w +J e n —we N . 



k=—nj=—n j,k,l=—n 

Using the shift property of the Discrete Fourier Transform we write 

n 

E 

k=—n 

where the * denotes the complex conjugate. Finally we obtain 

.. n 1 n 

TV N 2 ^— ' 

j,k=—n l=—n 

We notice that each of the terms in the above summation is positive. Indeed 
for all / A' 1 is real and symmetric positive, hence 

WA^'W* = tReiw^A^JReiw 1 ) + t kn(w l )A> lN ' l Im{w l ). 
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The linear term — J2 7 j=- n ( v '' ~ c M ,c mJV ) writes (w ,^). We conclude 
that the integrand in the definition of T2(n N ) is equal to l/(2cr 2 ) times 

i n 9 
i,^^i^* + l(,I,o, c ^) + || c ^f. (56) 

l=—n 

In order to show that this expression is bounded below, it is sufficient to 
show that 

^WA^w' + liw ,^), (57) 

is bounded below, where we have made use of the fact that w° is real. Let 
'° = M D** t O fl , where is diagonal and is orthonormal. We 
define X =* w°, so that (J37]l is equal to 

^XZ^VldT+i + ^ JV )^ 1 X + ^fof ,c^)X t , (58) 

t=o 

where Of is the t-th column vector of 0^ N . In order that (1581) is bounded 
below, we require that the coefficient of X converges to zero when D 11 does. 
The following lemma is sufficient. 

Lemma 22. For each < t < T, 

(cf N ,ofr<J^Df. 

X 5 1/— It 

Proof. If J — the conclusion is evident, thus we assume throughout this 
proof that J ^ 0. It follows from the definition that 

n 

m=—n 

Expressing f,m in terms of the matrixes ,k we write 

n 

K» N >° = 6 2 OZ T+l *OZ T+1 + A 7V (m, k)M^ N ' k . 



k,m=—n 



Since D*jf = *of 'k^^of , we find that 



d£ = e 2 (oz T+1 ,of) 2 + A N (k,m) t 0fM^ N ' k 0f. 

k,m=—n 
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We introduce the matrixes (L M ' )k=-n,-,m where for 1 < s,t < T, 

L£' k = ' k - = [ (yl, - -^{yti ~ K-i) V N (dy) 

Jt n 

where = ±cf N '. We define L$' h = if s = or t = 0. 

These matrices have the same properties as the matrixes M^^, in par- 
ticular their spectral representation (L M ' l )i=- n ,- ,n is positive. Using this 
spectral representation we write 

1 n 

Df = 9 2 {OZ T+1 ,ofr+A N (0,0)(c",of ) 2 +^ Yl ^ N (0-iyofL» N > l of, 

l=—n 

and since A Ar (0, — /) is positive for all / = —n, ■ ■ ■ ,n and 'Of Z/ A ,l O^ is 
positive for alH = 1, • • • , T, we have 

and the conclusion follows from assumption (jSJ). 



□ 



We may use the previous lemma to obtain a lower-bound for the quadratic 
form (1581) . We recall the easily-proved identity from the calculus of quadratics 
that, for all ieK, 

2 , b 2 

ax + 2bx > . 

a 

We therefore find, through lemma [22], that (1581) is greater than or equal to 
J 2 A ^ ^„„A J 2 



(59) 



(T + l)a 2 + g Df ) = -J- ((T + l)a 2 + Trace •«>))) . 

Since = and 

= ^ 2 40Z r+1 'OZ T+1 + ^ k{k,m)M 

m=— n 

it follows that 

Trace (K^' ) < (T + 1) (9 2 + A smn ) . 
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Putting all this together we find that the integrand of T 2 (fi N ) (we denote 
this (p N further below) is greater than —f3 2 , where 

= (T + 1)J 2 , 2 2 A , 
2a 2 A min V ; 

Note that we have 'recollected' the factor of l/2a 2 . This is a 'universal' con- 
stant which depends only on the model parameters and not on the particular 
measure fi. 

We then have the following proposition. 
Proposition 23. T 2 (fi N ) is lower-semicontinous. 



Proof. We take the integrand of fl55|) and 'shift' it up, so that it is positive. 
That is, we define 

V j=—n 



n n 



in (6i) 



N 

k=—n j=—n 



which, as we have just proved, is greater than or equal to zero. We define 
(f) N ' M (ljL N , v) = l BM (J) N (fi N iV), where v e B M if N~ l ^ n = „ n ||^'|| 2 < M. We 
also define 

rf(^)= / <f> N > M (^ N (dv)-p 2 . 

JS N 

Suppose that — > with respect to the weak topology. Observe that 



<f> N ' M {v N ,v)(i N (dv)- / <j) N > M (fi N ,v)ix N (dv) 



|rf(/^)-rf(/^)|< 

J^AT =k J sN =k 

We may infer from the above expression that T^/i^) is continuous (with 
respect to fi N ) for the following reasons. The first term on the right hand side 
converges to zero because <f) N ' M is continuous and bounded (with respect to 
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v). The second term converges to zero because <f> ' (fj, ,v) is a continuous 
function of fi N , see proposition [T71 

Since Y^{fi N ) grows to r 2 (/^ iV ) as M — > oo, we may conclude that r 2 (/x iV ) 



We define T(/x 7V ) = T 1 (/i Ar ) + r 2 (yU 7V ). We may conclude from proposi- 
tions [19] and [23] that T is measureable. It thus follows from definition [3] and 
proposition [10] that 

Corollary 24. 27je Radon- Nik odym derivative of H N with respect to R N is 
given by 



Proof. This is a matter of applying lemma in the case of 
Z = (G M ~ n , ■ ■ ■ , > n ), a = ^2 (t> ~ n , • • ■ , f n ), and b = ^, using the expres- 
sion flUD for T^/i^), and the fact that the measure yL N E Ml s (T N ). □ 

5 The large deviation principle 

In this section we prove the principle result of this paper (Theorem [2]), that 
the image laws U N satisfy an LDP with good rate function H (to be defined 
below). We do this by firstly establishing an LDP for the image law with 
uncoupled weights (R N ), and then use the Radon-Nikodym derivative of 
corollary [241 to establish the full LDP. Therefore our first task is to write the 
LDP governing R N . 

Let fi G Mi s {T % ). The Kvillback-Leibler divergence, noted J (2) (/A P m ), 
of [x N with respect to P® N = (P 2 )^ is defined as 



is lower semicontinuous with respect to ji 



□ 





d p®N 
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if fx N is absolutely continuous with respect to P®^, and I^(fx N , P®^) = oo 
otherwise. The process-level entropy of \i with respect to P z is defined to be 

I®(p,P*) = i im l/P)^, p®"). (63) 

TV— >oo iV 

P w is governed by the following large deviation principle [2~Tj [2]. If F is 
a closed set, then 

Em iV^logP^P) < - inf 7 (3) (//,P E ), 

and for all open sets O 

lim TV" 1 log P^(O) > - inf J {3) (/i,P z ). 

V-s>oo 

We note the following two properties of I^ 3 \ 

Lemma 26. is a good rate function (i.e. its level sets are compact). In 
addition, the set of measures {R N } is exponentially tight. This means that, 
for all < a < oo, there exists a compact set K a C M^ S {J~ 7L ) such that for 
allN 

lim" N~ 1 \ogR N (Kl) < -a. 

Proof. The fact that J {3) is a good rate function is proved in Ellis [23]. In 
turn, a sequence of probability measures (such as {R N }) over a Polish Space 
satisfying a large deviations upper bound with a good rate function is expo- 
nentially tight [IS]. □ 



Before we move to a statement of the LDP governing U N , we prove the 
following relationship between the set £2 (see definition H|) and the set of sta- 
tionary measures which have a finite Kiillback-Leibler information or process 
level entropy with respect to P % . 

Lemma 27. We have 

it* G .M+ S (T Z ), / (3) (/i,P s ) < 00} c E 2 . 

Proof. Let /x G Aif s (T z ). We use the classical result that 

I®(ji N ,P* N ) = sup (f <pd(i N -log [ exp(v?) dP®A . 

vec b (T N ) \Jr N Jr N J 
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We let p(y) = EL-JW ^ V = ap^f-^x^)), . . . , ^(f-\x n ))), 
where ^ is given in §FZ§ and a > 0. The function Pm( x ) — p( x )l<p{x)<M 
is in Cb{T N ), hence for all a > 



a [ p M dp N < log / exp(ap M ) dP m + J (2) (p N , P®? 

J T N J T N 

According to proposition [TJ P o ~ jV(0t+i, cr 2 IdT+i). Hence, as soon as 
1 — 2aa 2 > 0, we obtain using an easy Gaussian computation that 



log / exp(ap) dP m = _ ^E±jj i og (l - 2aa 2 ). 
By dominated convergence, letting M — > oo, we write 

a [ pdp N < log / exp(ap)dP® N + lM(ii N ,P® N ). 

J T N J T N 



Hence, since J rN pdp N = -/VE^[||?; || 2 ], we have 



(T j_ i ) tP>(ij n P® n ) 



al£ v u 1 < — log 1 - 2aa" + 



2 ' iV 

By taking the limit iV — > oo we obtain the result. □ 

We are now in a position to define what will be the rate function of the 
LDP governing U N . 

Definition 6. Let H be the function Mf s (T z ) -fRU {+00} defined by 

f +00 if J( 3 )(/i,P s ) = 00 
W \ I^(p,P z ) -r(/i) otherwise. 

Note that because of proposition |2"U1 and lemma ¥2T\ whenever 7®(/i,P K ) 
is finite, so is r(/i). It also needs to be noted that, for all N and x G T N , 
p N (x) G £ 2 . 

Our proof of the principal result, Theorem [21 of this paper will occur in 
several steps. We prove in sections 15. II and T5. 31 that 11^ satisfies a weak LDP, 
i.e. that it satisfies ( I17jl when F is compact and (118j) for all open O. We also 
prove in section 15721 that {n^} is exponentially tight, and we prove in section 
15.41 that if is a good rate function. It directly follows from these results that 

satisfies a strong LDP with good rate function H (19] . Finally, in section 
[6] we prove that H has a unique minimum which p converges to weakly as 
N -> 00. 
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5.1 Lower bound on the open sets 

We prove the second half of proposition [2] 
Lemma 28. For all open sets O, 

lim N' 1 log 11^(0) > - inf H(ji). 

Proof. From the expression for the Radon- Nikodym derivative in corollary 
[24] we have 

U N (0) = [ exp (NT(ti N )) dR N (n). 
Jo 

If /i G O is such that /^(/i, P z ) = oo, then H(fi) = oo and evidently 

lim N- 1 log U N (0) > -H(fi). (64) 

JV->-oo 

We now prove ( 1641) for all fi G O such that I^ 3 \fi, P % ) < oo. Let e > and 
Z^(fi) C O be an open neighbourhood containing fi such that 
vai^zN {fj) r(7 Ar ) > r(/i 7V ) — e. Such {Z^(/i)} exist for all N because of the 
lower semi-continuity of T((j, N ) (see proposition [23]) and the fact that the 
projection \x — > fi N is clearly continuous. Then 

lim AT 1 log 11^(0) = lim A^" 1 log / exp(NT('y N ))dR N ('y) 

N-too N-^oo JO 

> lim JV -1 log \R N {Z^) x inf exp(iyr( 7 7V )) ] 

> -J (3) (u,P z ) + lim inf rfr^) 

> -J (3) (/i,P z ) + limr(/)- £ 

= -/( 3 )(/i,)p z + r( / i)- e . 

The last equality follows from lemma [18] and proposition [20] Since e is 
arbitrary, we may take the limit as e — > to obtain (1641) . Since (|6~4"1) is true 
for all /i G the lemma is proved. □ 

5.2 Exponential Tightness of II 

We recall that if fi G A^^ S (T Z ) but /i £ £ 2 , then /( 3 )(/i,P s ) = r(/i) = oo. 
Otherwise, 1^ and T satisfy the following affine inequality. 
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Proposition 29. There exist constants a > 1 and c > such that for all 

»eMt,(T*)n£ 2 , 

< (/«(,, n + c) 

a 

We have (from ([63])) that 

/( 3 )(u,P z )= lim AT-i/C 2 )^^^). 

jV-»00 V 7 

We recall that J (2) may be expressed using the Fenchel-Legendre transform 

as 

= sup ( / 4> N (x)/i N (dx)-\og [ exp((f) N (x))P® N (dx)) , (65) 

where <p N is a continuous, bounded function on T N . We choose a specific 
function 0^ to be N times the sum of r 1 (// 7V ) and the integrand of T 2 {fi N ) 
(i.e. f lo^]) ) to which we add the constant = 0\ + /3 2 to make it positive. In 
detail, <fi N {x) = 4> N (v), where = ^/(f~ 1 (x : ')) and 

\j,k=—n 

+ 2 ^( C ^,^)-iV||c^|| 2 ), (66) 

j=-n / 

and (3 = /3i+/5 2 - 0i and 02 are defined in equations ( 1511) and ( 160]) . respectively. 
4> N is continuous but not bounded in general. Hence we multiply it by 

]-\\v\\ 2 <m(x) to obtain 

4>m{%) = <p N (x)l\\vp<Al(x), 

which is continuous and bounded. 

It follows from (1651) that, for all a > 0, 

a [ K(^ N (dx) < log / exp (a4&(x)) P m (dx) + I^^ N , P® N ). 

J T N J T N 

(67) 
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We wish to use the dominated convergence theorem to prove that (j67p holds 
in the limit as M — > oo. 

We may do this using the following lemma, 

Lemma 30. There exists a positive constant c < oo and a > 1 such that, 
for all N, 

exp(a(f) N (x))P® N (dx) < exp(iVc + aN/3). 



Proof. We find from proposition [T] that 

exp(a<f) N (x))P m (dx) = [ exp(a<J) N (v))P® N (dv). 

T N J S N = 

We use the spectral representation of ( 1561) to rewrite the third term in ((66 

as 

1 n 

£ WA" > l w l * + 2(cf, w°) + N\\c^\\ 2 , 

l=—n 

where 

n 

W = y, w e N I = ~ n i ' ' ' j n ; 



k=—n 

,N 



and w k = v k — c M , k — —n, ■ ■ ■ ,n. Because the sequence (v k — c M )k=-n-n 
is real, the real part (Re(w'))i = _ n ... n of the sequence {w l )i = ^ n ... n is even and 
the imaginary part (Im(iZ>'))/ = _ n ... n , odd. We perform the bjective affme 
change of variables in S N 

h:(v~ n ,--- ,v n )^(y- n ,--- ,y n ), 

where, for s G [0, T], 

V2Re(w- k ) k = -1, ■ • • , -n 
y k s = { w° s k = 

V2lm(w k ) k = 1, • • • , n 

Moreover, the sequence (^4 M ' fe )fc=- n ,-,n is symmetric even, which implies 
that 



j2 wa^'w 1 * = y^v- 



l=—n l——n 
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We thus find that 



1 / 1 



^(h-^y)) = NTM + Nf3 + —[- VA" 'V + 2{c fl ,y ) + N\\c»\\ 2 + 



2a 2 \ N 



l*l=i / 



N 

Under h, it is easy to check, using the properties of the Discrete Fourier 
Transform, that 



n 1 / n 



,i|| 2 _i_ II „o 



||y° + ^||' 



j=-n 



Jil=i 



It follows that 



P m oh-\dy) = V27rNa 2 



-N(T+1) 



X 



exp 

Hence we write 

exp(a0 7V (t;))P 0Af (^) 

5 iV — 

where 
G 1 = (V2ttNct 



\ \j\ = l / / t=0j=-n 



S N 



expiaNiT^) + /?)) x G 1 x G 2 , 



-(T+l) 



exp 



2iV<7 2 

T 



t=o 



and 



G 2 = ( V2tiN(t 2 



-(jV-l)(T+l) 



S ( W -i) 6XP 2iVa 2 



^aVi^V-ll^'ll 2 

b1=i 



n T 



.7 1=1 t=0 
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We assume that a > 1 is such that (1 — aa) > 0, where a is an upperbound 
for the eigenvalues of A 11 given in Lemma [2TJ We find 



Go < ( V2irNa 2 



-(jV-l)(T+l) 



s( n-x) 6XP 2iW 2 



J>a-l)||y|| 2 

lil=i 



n T 



lil=i *=o 



lil=i 



cxp 



2iVV 



Ml 2 



where 



F J ~ Af T+1 (0 T+1 ,Na 2 ld T+1 ), \j\ = 1,- •■ ,n. 
The application of lemma |6] yields 



exp 



aa 



\Y 3 \ 



'1 — aa) 



-(T+l)/2 



and 

Similarly 



2iV> 2 

G 2 < {l-aa)^ N ^ T+1 ^ 2 . 



2a 2 



aa ,w r0M 2 , a - 1 /v o 



2iW 2 



|F u || 2 + ^(r u ,c^ ) 



(7 



where 



Y° ~ N T+1 (0 T+U Na 2 Id T+1 ). 
Another application of lemma yields 



G x < exp N- 



-niic^ii 2 



2a 2 



n fT+iy 2 IK II 2 

x (1 - aa)" (T+1)/2 x expiV-^- — 1 " " . 



2(1 - aa)a 2 



Since 
we have 



1; 



ri(/x w ) + /3</3. 
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Putting all this together we obtain 

f exp(a<p N (x)) P® N (dx) < exp (Nc + aN/3) , (69) 
Jr N 



where 



T + l a(a — 1)(1 — a) „ ,.jv ii2 

•— — logl-aa + \ A ^ c" 2 >0. 

2 2cH 1 — aa 



□ 



We may now conclude the proof of the proposition. 



Proof. We take M — > oo and apply the dominated convergence theorem to 
( 1671) . noting that (j)^ grows to <fi . We thus find that 



a 



<p N (x)fi N (dx) < log / exp (fl^(x)) P^dx) + I (2 >0A P 07V ). 



This and (jBSJ imply that, for all TV, 

aiV (r(^) + /?) < iVc + aiV/3 + J (2) (/A P 0JV ), 

as required. We divide both sides by aN and let N — > oo to obtain the 
required result. □ 

Proposition 31. The family {11^} is exponentially tight. 

Proof. Let B G £(.M^(T Z )). We have 

U N (B)= [ expNT(fi N (x))P m (dx). 

Through Holder's Inequality, we find that for any a > 1 such that 1 — aa > 0: 

IL N (B) < R N (B)^ ( [ exp (aNT (fi N (x))) P® N (dx) 

Now it may be observed that NT(fi N (x)) = (x) — N/3, where <& N is defined 
by fl66|) . It therefore follows from lemma 1301 that 



U N (B) < R N (B)^J exp 



(70) 



47 



By the exponential tightness of {R N } (as proved in lemma |2~6T) . for each 
L > 0, there exists a compact set Kl such that 

lim iV" 1 \og(R N (KD) < -L. 



It may be seen that if we choose 



B — K a f T i c \ 



then we obtain from f!7Up that 



lim N- 1 \ogIi N (B) < -L 

AT— 5>oo 

as required. □ 



5.3 Upper Bound on the Compact Sets 

In this section we obtain an upper bound on the compact sets, i.e. the first 
half of theorem [2] for F is compact. Our method is to obtain an LDP for a 
simplified Gaussian system (with fixed A u and c v ), and then prove that this 
converges to the required bound as v — >■ ji. 



5.3.1 An LDP for a Gaussian measure 

We linearise Y in the following; manner. Fix v G -M^ S (T S ) and assume for 
the moment that Let 




where A"> N > k , k = -n, ■ ■ ■ , n is the fcth (T + 1) x (T + 1) block of the 
N[T + 1) x N(T + 1) symmetric block circulant matrix 



K»> N (a 2 Id N(T+1) + K»' N )-\ 

and K U,N is the N(T + 1) x N(T + 1) covariance matrix of the Gaussian 
process (G"' ,J )j = _ n! ... )n defined in section 13.1.21 
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Let us also define 

r?M = -^ lQ g det (W+i) + ^**") , 

and let 

rV0 = r?M + rZ(A*")- 

We let = liniAr^oo T^(fi N ), (for a = 1 or 2) and find, using the seecond 

identity in proposition [2Q1 that 

-2*^(0)^ + W^O)^ + 2(cV^> - IKH 2 ) , (72) 



where = K=[v°], and v^ 1 is the spectral measure of the process (\I/ o 
f^ 1 (x k ))k£'z given in (130]) . The spectral measure exists as long as /i £ £2, in 
which case the above is finite. We recall that : denotes double contraction 
on the indices. 

Similarly to lemma [TBI we find that 

Jirn^Tf (1/) = -i-y Mogdet Hd T+1 + -L#»^ rfu; = (73) 

For /1 £ £ 2 , we define = J (3) (/i,P s ) - r"(/x); for // £ E 2 , we define 

T^/i) = r^(/i) = 00 and H u (fi) = 00. In fact it will be seen that H v is the 
rate function for the Gaussian Stationary Process Q v to be defined below. 
We define the following measure over S N . For B £ B(S N ), 

Q^ N (B) = J exp (AT^/i^))) E® N (dv). (74) 

This defines a law Q U)N over according to the correspondence in definition 
[2 We find that 

g u ' N (B) = (det (ld N{T+l) + ^ ,JV )) 2 x 

y ex p 2^2 ( E v - v +j - 0+ 



\J,k=—n 



2 (c",^) -iVHcl 2 jjf^tfo). (75) 
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We note c u,N the N(T + 1) -dimensional vector obtained by concatenating N 
times the vector c u . We also have that 

^(Id N{T+1) - A V,N ) = (a 2 ld N(T+1) + K»> N )-\ 

Thus, through proposition [Tj we find that 

g»> N (B) = (27r)-^ ^det (J^(Id N{T+1) - A»> N ^j 

exp (v - c^) (ld N{T+1) - A U ' N ) (v - <?>») J] J] dv{. (76) 

j=-n t=0 

It is seen that Q U,N is an N(T + l)-dimensional Gaussian measure with mean 
c v ' N , inverse covariance matrix T-(Idjv(T+i) — A 1 ^), and covariance matrix 
^Id^r+x) + if 1 ^. Hence Q V ' N is in M^^). 

We may thus define the measure Q v of a stationary Gaussian process 

over the variables {vi}jez,s=o,-,T, with iV-dimensional marginals given by 
( 1761) . The corresponding infinite dimensional Gaussian measure Q u on S % 

has covariance operator <r 2 Id + K v and mean c 1 '. It may be observed that 
the spectral density of the covariance is <r 2 IdT+i + K v ■ 

Let U^ N be the image law of Q v under fi N , i.e. for B E B(Mf )S (S z )), 

1?> N (B) = Q u (jf e B) . 

Lemma 32. The image law Tl u ' N satisfies a strong LDP (in the manner of 
proposition [||) with good rate function 

K v (^) = I l3 %R z )-T v ^), (77) 

where 1^ : A^^ s (iS K ) — > R, U oo is defined analogously to ( )63l) . 

This result is deduced from [21 EI], see appendix|A] For I? E B(Mt yS {T z )), 
we define the image law 

U V,N (B) = Q v {fi N EB)= Q"^ E $ o f-\B)). 

It follows from the contraction principle that if we write H u (fi) := H" (p) , 
then 
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Corollary 33. The image law U U,N satisfies a strong LDP with good rate 
function 

H v (n) = I®(ji,P z )-T t '(j l ). (78) 
In particular, we note that / (3) (/i,P K ) = J (3) (/i, P z ). 

5.3.2 An upper bound for II ^ over compact sets 

In this section we derive an upper bound for IT^ over compact sets using 
the LDP of the previous section. Before we do this, we require some lemmas 
governing the 'distance' between T u and V. Let 

C U N = sup {\\A vM ' 1 -A v > M %\\K vM > 1 -K V > M > 1 \\}, (79) 

M>N,(2\l\+l)<M 

where we have taken the operator norm. Recall that K U,M and A U,M are 
defined in Section 15.3.11 

Lemma 34. For all v £ .M+ S (T Z ) , C U N is finite and 

C V N as N ->• oo. 

Proof. We recall from proposition [291 that K"™ (oo) converges uniformly (in 
oo) to K" t (oo). The same holds for K^ M '\ because this represents the partial 
summation of an absolutely converging Fourier Series. That is, for fixed 
oo = 2vr/ A/ /M, K"i M ' lM K v 8t {uo) as M -> oo. The result then follows from 
the equivalence of matrix norms. The proof for A u is analogous. □ 

Lemma 35. There exists a constant Cq such that for all v in jW\J(T % ), all 
e > and all \i £ V e (u) fl £2, 

|r(/0 - r> w )| < c (c» N + e )(i + m\\v°\\ 2 })- 

Here V £ {y) is the open neighbourhood defined in proposition\Tl\ and fi is given 
in definition^ 



51 



Proof. We firstly bound r\. 

|iWi?(i/)|< 

1 n 

— 2 | lo g det ( Id r+i + a -2 ^' 1 ) - logdet (ld T+1 + a- 2 K» N ^ 

l=—n 
1 n 

+ ^ E | lo S det ( Idr +! + ""^""'J - lo S det ( Id T+i + ^i^) . 

l=—n 

(80) 

It thus follows from proposition [T7] and lemma EH that 

IrxCAi^-lfMl <C*(C^ + e), 

for some constant Cq which is independent of v and iV. 

It remains for us to bound Yi- The proof uses a slightly modified version 
of the spectral representation used in the proof of lemma [30j We perform 
the bijective affine change of variables in S N 

h:(v- n ,..- ,v n )^(y- n , ■■■,y n ), 

where, for s G [0, T], 

( V2Re(v; k ) k = — 1, • • ■ ,—n 
y k s = lv° s k = . (81) 

[ \/2Im(v k ) k = 1, • • • ,n 

(v k ), k = —n, ■ ■ ■ , n is the discrete Fourier transform of the sequence (v h ). 
This allows us to write the integrand of r^/x^) (up to the factor l/2a 2 ) 

as 

1 n 9 
N 2 ^— ' N 

l=—n 

Similarly we write the integrand for r 2 (/i 7V ) 
1 n 9 

_L y yi^y + a (c m _ i^»v, y °) + va^-v - h^ii 2 . 

iV 2 z — ' TV 

Z=— n 
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Hence we have 



|r 2 (^)-r^)|< 

£ ii^'-^'ii^ 

\ Z=— n / 

where d v ^ = d 1 - c u + A^'V - A^'Vand e u ^ = ^i^'V - ||c^|| 2 - 
V^'-^'V+llcfll 2 . Here || A^'-A^'H is the operator norm but ||d„ )/Lt || is the 
vector norm. We may bound the coefficients through the following identities. 



7^,1 7v»,l 
A st ~ SI st 



< € 



It was proved in proposition [T71 that for all < s,t < T 

and \c u s — < e. We may thus infer that ||A mJV '' — A^'H < (T + l)e and 
\\c u — c M || < (T + l)e. Furthermore, we have from lemma EH that \\A V ,l — 
A V ' N > 1 \\ < C V N , and || c v || is bounded by TJ 2 for all v. 
We thus observe that 

lli^-i^'ll < \\A» N ' l -A» N > l \\ + \\A» N ' l -A»> N > l \\ l = -n,---,n, 

\\duj < V - c"|| + ||A>" V - A"" V|| + || (A v > n >° - A""' ) c% 

I I 



< I^A^'V - l c v A vN ^c v \ + |V (A 1 ^' - A 1 



It is evident from the above considerations that each of the above terms 
is bounded by C*[C V N + e) for some constant C*. The lemma now follows 
after consideration of the fact that f gz ||t; fe || 2 /z(cfo) = E=[||t;°|| 2 ], ||y°|| 2 < 

NY^k=-n \\ vk \\ 2 an< i' because of the properties of the discrete Fourier trans- 
form 

n n 

Z>'H 2 = tf £ \\A 2 . (82) 

l=—n k=—n 

□ 

We are now ready to begin the proof of the upper bound on compact sets. 

Proposition 36. Let K be a compact subset of A^i, s (T a ). Then 
lim N- 1 \og(]l N (K)) < - inf K H. 

TV— >oo 

Proof. Fix e > 0. Let V e (u) be the open neighbourhood of v defined in propo- 
sition[T71 and let V e {y) be its closure. Since K is compact and {V £ {v)} v& k is 
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an open cover, there exists an r and {^i}[ =1 such that K C U[=i We 
find that 



lim at 1 log [\JvMnK 



\i=l 



< sup lim N- 1 log (n^ (F £ (^) n K)) . 

l<i<r N ^°° 

It follows from the fact that /t G £2, lemma [35] and the definition of U N 
that 



U N (VM) ^K)< [ exp ( iVr^(/i Ar (x)) + 

iA JV (^)6Vi(^)n^ \ 

NC (e + C%) (l + jjY, Ml 2 )) (83) 



where w- 7 = 1 (x^)). From the definition of Q U,N in ( I74|) and Holder's 
Inequality, for p, g such that ^ + - = 1, we have 

n N (v e (vi) n k) < (Q^ N (ji N (x) g K(^) n k))*d\, (84) 



where 
D = 



fi N {x)eV £ {u t )nK 



exp ^iVC (£ + Cft) ^ + ^ E Ml 2 )) 



.7 = 

expgiVCo(£ + Q)x 



/ exp UCo(£ + ^) IM| a J ) ^(dw). 

^(„)eM>o/-i(Vi(^)n^) \ \j = _ n J J — 

We note from lemma [14] that the eigenvalues of the covariance of Q Vi ' N are 

upperbounded by a 2 + pK- Thus for this integral to converge it is sufficient 
that 

This condition will always be satisfied for sufficiently small e and sufficiently 
large N (since C 1 ^ — > as N — > oo). By corollary l33l 

lim" iV" 1 log (Q^f^fx) G t4(z/i) n JO) < - inf H Vi U). (86) 
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We know that Q Ui,N is Stationary Gaussian with mean c Ui and covariance 
a 2 ld N (r + i) + K VuN . We apply lemma [6] to find 

/ expqC (e + C%) ( V \\v j f) Q^' N (dv) < 

^(«)6®o/-l(V e (i/0nK) \ i= _ n / — 

(det ((1 - 2qC (e + C%)a 2 )ld N(T+1) - 2qC (e + C%)K Vi > N ))-* x 
exp (2C 2 q 2 ((e + C^) 2 )\l N{T+1) ^)B{l N{T+1) c^) + NqC (e + Q)ll^f ) 

where 1jv(t+i) is the N(T + 1) x (T + 1) block matrix with each block Idx+i 
and 

B = (a 2 Id^ (T+1) +^^)((l-2C , g(£+C , ^)a 2 )Id w{T+1) -2C'og(£+Q)^ 1/l ' Ar )" 1 

is a symmetric block circulant matrix. 

We note B k , k = —n, ■ • ■ , n its T x T blocks. We have 

\k=-n J 

where _B° is the Oth component of the spectral representation of the sequence 
(B k )k=- n ,- ,n- Let v m be the largest eigenvalue of B. Since (by lemma E]) the 
eigenvalues of B subset of the eigenvalues of B, we have 



, L N(T+1) 

c Vi )B(l N{T+1) c^) < Nv n 



c 



fill 2 



From the definition of B and through lemma dH we have 



, (t 2 + Pk 

V m < 



l-2C q(e + C%)(a 2 + p K ) 
Hence we have, since ||c^|| 2 < TJ 2 

exp (2C 2 (q 2 (e + C^) 2 \l N{T+l) c^)B(l N{T+l) c^)) < 

2C 2 q 2 (e + C%) 2 (a 2 + p K )J 2 - 



exp NT x 



l-2C q(e + C%)(a 2 + p K ) 
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Since the determinant is the product of the eigenvalues, we similarly find 
that 

(det ((1 - 2C q(e + C^)a 2 )ld N{T+1) - 2C q(e + C%)K Vi > N ))~^ < 

(l-2C q(e + C%)(a 2 + p K )) 



N(T+1) 



Upon collecting the above inequalities, and noting that 1 1 1 1 2 < (T + 1) J 2 , 
we find that 

D<exp(Ns%(q,e)) t (87) 

where 

*£(g, e) = (T + 1) f-i log (1 - 2C q(e + C%)(a 2 + 

, 2C 2 g^ + Q)V + PA')</ 2 , ^ , , ^ m / 1 
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' 1 - 2C q(s + Q)(a 2 + PK ) ' ^ + ^ ItTT + J 
We let s(q,e) = lim s^(g,e), and find through lemma I3H that 

AT— >oo 



Kg, e) = (T + 1) ^-i log (1 - 2C qs(a 2 + 



^^ + PK )J^ (1 +J2 



I - 2Coqe(a* + p K ) \T + 1 

Notice that s(q,e) is independent of z/j and that s(q,e) — > as e — > 0. Using 
(El, ([86]) and (E7D we thus find that 



limiV- 1 bg(n JV (A')) < sup -- inf H v Hp) - -s(q, e). 
N^oo i<j< r p neKnVein) q 

Recall that H u (p) = oo for all p ^ E^. Thus if K D £2 = 0, we may infer that 
lim A r ~ 1 logfn^f/^)) = — oo and the proposition is evident. Thus we may 

N— >oo 

assume without loss of generality that inf ^ K H Vi (p) = inf^gxn^ H Vi (p). 
Furthermore it follows from proposition [37] (below) that there exists a con- 
stant Cj such that for all p e V £ (ui) R £"2, 

*r«M > /( 3 )(/i,P z ) - r(/i) - c je (i + / (3) (/i,P 2 )). 



5(3 



We thus find that 



lim N- 1 hg(U N \K)) < 



- - inf (/( 3 )(/i,P a )(l - Cje) - Tfjj)) - ^l£) + t Cl , 
p Kn£ 2 Q V 

We take e — > and find, through the use of proposition |29l that 
Urn" iV" 1 logfn^fK)) < -- inf (l {3) (li : P % ) - T(u)) . 

N^-oo p K 

The proof may thus be completed by taking p — y 1. □ 

Proposition 37. There exists a positive constant Cj such that, for all v 
in MX^{J~ % ) H S 2 , all e > and all \i e K(z/) n £ 2 (Wiere £/ie 
neighbourhood defined in proposition [77|), 

|r"M - r^o*)| < c /£ (i + /^(/i, p z )) . (88) 

The proof is very similar to that of proposition [291 and we have therefore 
left it in the Appendix. 



5.4 H is a good rate function 

Lemma 38. H(fi) is lower- semi- continuous. 

Proof. Fix fi and let (/z m ) m >o converge weakly to fi as m — > oo. We let (fi Prn ) 
be a subset such that lim H (/i m ) = lim H(^ Pm ). Suppose firstly that 

m->oo m->oo 

lirrU( 3 )(/i pm ,P s )=oo. (89) 
From proposition [29] we have that, if fi Pm G £2, then 

H(fi Pm ) > (l — 7) -^ ( ' 3 ' ) (/%>m) ~~ ~j where a > 1 and c > are constants. 
Otherwise, if \i Vm £ £2 then (through lemma [27]) H(fi Pm ) = 00. In either 
case, we find that lim if (// Pm ) = lim H(ix Pm ) = 00, so that in this instance 

i7 is lower- semicontinuous at /i. 

In the second instance, we assume that ( 1891) does not hold, so that there 
exists an M such that for all m > M, {1^ (/x Pm , -P 2 )} is upperbounded (and 
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by lemma 123 fJ> Pm 6 £2)- We then find that 

!im#(/vJ= lim (/ {3) (/i Pm)J P S )-r(/i Pm )) 

> hmfP(/i p J+ hm^-r)^). 

Recall that r(/x Pm ) = r Mpm (/i Pm ). It follows from proposition 1371 and the 
boundedness of 1^ (/i Pm ) that the second term is zero. However is 
lower-semi-continuous [2], which allows us to conclude that hm if M (/x Pm ) > 

H 11 ^) = H(fi) as required. □ 

Because {II } is exponentially tight and satisfies the weak LDP with rate 
function H(fi), the following corollary is immediate [20j Lemma 2.15]. 

Corollary 39. H(fi) is a good rate function, i.e. the sets {fi : H(fi) < 6} 
are compact for all 5 G R + . 



6 The unique minimum of the rate function 

We first prove that there exists a unique minimum /x e of the rate function, 
before proving that 11^ converges weakly to 5^. We finish by providing 
explicit equations for fi e which would facilitate its numerical simulation. 

Lemma 40. For e Mf^iT^), H^(v) = if and only if v = . 

Proof. Using the correspondences in section 15.3.11 it suffices to prove that 

W"{v) = if and only if v = Q^ 1 . Let fc 6 s be the Fourier Coefficients 

_ 1 

of (ff 2 Id + r>) ~\ i.e. 

W k = J exp (iku) (a 2 Id T +i + K M (w))"' du. 

This is well-defined because o" 2 Ht+i + K^(u) is symmetric and its eigenval- 
ues are strictly positive. Let : 5 Z — > S % be the map v — > [r^{v)) k = 
EZ-00 W>\v k - 1 - and let ~ Af T+1 {0 T+ u I<W 
The result in [21] stipulates that 

g»(g) = j(3) (j/o(r^\g) . 
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In turn a contraction principle [23] dictates that 

with equality if and only if u o (r^y 1 = P^. We note that P^ = o 

(r^)" 1 , and that if H<*{y) = J (3) (t~ m ) _1 ,£J) < °° then g is absolutely 

continuous with respect to (as noted in Section f5 . 3 . 1 p . The lemma now 
follows from the fact that r M is one-to-one on the set of all measures absolutely 
continuous with respect to P^. □ 

Proposition 41. There is a unique distribution fi e G Mf s (l~' z ) which min- 
imises H . This distribution satisfies H(fi e ) = 0. 

Proof. By the previous lemma, it suffices to prove that there is a unique \x e 
such that 

= He- (90) 

Let Tt be the cx-algebra over T a generated by (a^)ies,r=o,-,t, and Tt_ the 
cr-algebra over S % generated by (yj:)ies r=o ••• t- We define the mapping L : 
Ml,(T*) ^ Mi,(T*) by 

H -> L(/i) = 

It follows from fl9]) that 

Qfa = fi, (91) 

which is independent of \i. 

It may be inferred from the definitions in Section 14.1. 21 that the marginal 
of L(n) = Q* 1 over jF t only depends upon the marginal of \i over Tt-i- 
This follows from the fact that Q^\t s (which determines Q^ L \r s ) is com- 
pletely determined by the means {c^;t = 0, . . . , s — 1} and covariances 
{K£tf;j eZ,«,«6 [0, s — 1]}. In turn, it may be observed from ( 136|) and (1591 
that these variables are determined by ^Ij^-i- Thus for any fi, v G .M^CT 2 ) 
and £ G [1, T, if 

then 
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It follows from repeated application of the above identity that for any v 
satisfying v\j a = /xf , 

L t (^ t =L(L t (u)) ITt . (92) 

Defining 

fi e = L T (v), (93) 

it follows from (192]) that /x e satisfies ( 1901) . 

Conversely if /x = L(fi) for some /x, then we have that /x = L 2 (u) for 
any v such that v\t t _ 2 = ^\t t _ 2 - Continuing this reasoning, we find that 
/x = L T (v) for any v such that v\? a = fJ>\jr . But by (l9Tj) . since = /x, we 
have /jL\jt = fif. But we have just seen that any /x satisfying /x = L Tt 



where v\p = /xf, is uniquely defined by (l93~]) : which means that /x = /x e . □ 
Theorem 42. Il^ converges weakly to 5^, i.e., for all $ G Cb(Mf s (T' z )), 



lim / $(/x 7V (x))Q 7V (rfa;) = $(Ai e 

N-^OO J q-N 



Proof. The proof follows directly from the existence of an LDP for the mea- 
sure n^r, see theorem [2J and is a straightforward adaptation of the one in 
Theorem 2.5.1]. □ 



We may use the proof of proposition |4T] to characterize the unique mea- 
sure /x e such that /x e = Q^ e in terms of its image /x e . This characterization 
allows one to directly numerically calculate /x e . We characterize /x e recur- 
sively (in time), by providing a method of determining [i e \T t m terms of 
/x e |j- t l . However we must firstly outline explicitly the bijective correspon- 
dence between \i e \T t an d ^e\T t i as follows. For v G S, we write \l/~ 1 (f) = 
(tf-W.-,* -1 ^). We recall from © that * _1 (?;)o = ^Oo)- The 
coordinate \l/ _1 (f) t is the affine function of v s , s = 0---t obtained from 
equations (TTT1) and (IT2"1) 



t-i 



8=0 



-7* - 1 



Let ^^(i;) : R m -> R m be such that 

*${v , ...,vt) = (*o '(vo), ^Hi, . . . , 
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where w G S is such that w s = v s for < s < t. With the same notations as 
in definition [3] we have /i|j- t = /i|j- t o iff^ o 

In the course of the previous proof we saw that /i e | jr Q = n® % and \i e \T a = 
A/"(0, a 2 )® 2 , which gives us the first step in our induction. It remains for 
us to explicitly outline how we determine ii e \T t from [i e \F t -i f° r eacn t > 1. 
We saw in the previous proof that both of these are Gaussian Processes. As 
was explained, it suffices for us to provide expressions for cf e and {i£^ t eJ , s = 
0, G %} in terms of [i e \T t -\ (note that is symmetric). The 

other components of the mean and covariance of [i e \F t are the same as their 
analogues in \i\F t _ x - The mean is given by 



■/]0.1[< JR f V J 



when: ^ is the marginal distribution over one neuron. 

The formula for K^'i can be obtained from equations and (1TT|) . 
Indeed, we have 

oo 

J^>i = # 2 OZ T+1 *OZ T+1 + ^ A(j,l)M^' 1 , 

and 

K^ 1 = [ Vr-d^dfieiy) r, s > 1. 

If r = or s = 0, then M£' fc = 0. This can be rewritten, in the case of 
1 < r, s < t, as 

= [ (/ o ^(A-i) x (/ o ^^(d^). 
Here /^'^^(cfoW) is distributed as Mt((c{£_i), cjl^), K^ l) ), where 



-Me _ f,,Me Me \ 



7^6,(0,0 



A (t-i) 



and -K^Hi) is the i x t submatrix of j^ e,z composed of the elements from 
times to (t — 1). 
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One cannot in practice numerically calculate all of the K^ 3 at each time 
step, as there are an infinite number of neurons. However since A(j, k) must 
decay to zero as either j or k asymptotes to infinity, we strongly expect that 
if we only simulate N neurons, then the results will converge as N — » oo. 
We note that numerical simulation using the above procedure would likely 
be highly unstable as one would expect errors to accumulate exponentially. 
It is possible that numerical simulation of the spectral densities would be 
much more accurate. We will explore these questions further in a subsequent 
paper. 



7 Conclusion 

In this section we sketch out some important consequences of our work and 
possible generalizations. 

7.1 Important consequences 

We note that the LDP of Moynot and Samuelides [MJ [35] may be obtained 
from ours by stipulating that A(a, b) is nonzero if and only if a = b = 0. 
Their LDP may then be obtained by applying a contraction principle to our 
LDP through taking the 1-dimensional marginal of fi N . More generally, for 
any d G Z + one may obtain a process-level LDP governing the interaction of 
each neuron with its d neighbours by applying a contraction principle to the 
d— dimensional marginal of the empirical measure. 

We state some implications of our results (particularly theorem [2]). 

Corollary 43. For all h e C b {T 7L ), 

1 " 
lim — > 

N^oo N ^— ' 
i=— n 

Proof. It is sufficient to apply theorem|32]in the case where $ in Cb(Mi a (7^)) 
is defined by 

= / hd\x 

□ 



h(S l (x(N)))Q N (dx 



h(x) dfi e (x) 
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Since the proof of theorem 02] only requires the use of the rightmost 
inequality in the definition of the LDP, we can in fact obtain a quenched 
convergence result through the use of the Borel-Cantelli lemma. We recall 
that Q N (J, 0) is the conditional law of N neurons for given J and 0. 

Theorem 44. For each closed set F of M.'l s {T % ) and for almost all (J, 6) 

lim -j-log [Q N {JM» N G F)] < ~ ^H(fi). 

Proof. The proof is a combination of Tchebyshev's inequality and the Borel- 
Cantelli lemma and is a straightforward adaptation of the one in [MJ Theorem 
2.5.4, Corollary 2.5.6]. □ 

This result allows us to state the quenched analog to theorem l4*2l 

Corollary 45. For all $ 6 C b (.M^(T K )) and for almost all (J, 9) we have 

lim / §(fi N (x))Q N {J,d)(dx) = $(/x e ) 
7.2 Possible extensions 

Our results hold true if we assume that equation ([1]) is replaced by the more 
general equation 

l n 

^ = E^ + E J i*i) + ^ + ^-i, j = -n,...,n t = l,...,T, 

k=l i=—n 

where I is a positive integer strictly less than T (in practice much smaller). 
This equation accounts for a more complicated "intrinsic" dynamics of the 
neurons, i.e. when they are uncoupled. The parameters 7^, k = 1 • ■ ■ I must 
satisfy some conditions to ensure stability of the uncoupled dynamics. 

The extension to continuous time is certainly worth considering even 
though we expect it to be quite difficult. 

The hypothesis that the synaptic weights are Gaussian is somewhat unre- 
alistic from the biological viewpoint. In his PhD thesis [31], Moynot has ob- 
tained some promising preliminary results in the case of uncorrelated weights. 
We think that this is also a promising avenue. 
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Moynot again, in his thesis, has extended the uncorrelated weights case, to 
include two populations with different (Gaussian) statistics for each popula- 
tion. This is also an important practical problem in neuroscience. Extending 
Moynot 's result to the correlated case is probably a low hanging fruit. 

Last but not least, the solutions of the equations for the mean and covari- 
ance operator of the measure minimizing the rate function derived in section 
Inland their numerical simulation are very much worth investigating and their 
predictions confronted to biological measurements. 

7.3 Discussion 

In recent years there has been a lot of effort to mathematically justify neural- 
field models, through some sort of asymptotic analysis of finite-size neural 
networks. Many, if not most, of these models assume / prove some sort 
of thermodynamic limit, whereby if one isolates a particular population of 
neurons in a localised area of space, they are found to fire increasingly asyn- 
chronously as the number in the population asymptotes to infinityjj Indeed 
this was the result of Moynot and Samuelides. However our results imply 
that there are system-wide correlations between the neurons, even in the 
asymptotic limit. The key reason why we do not have propagation of chaos 
is that the Radon-Nikodym derivative of the average laws in propo- 
sition [3] cannot be tensored into N i.i.d. processes; whereas the simpler 
assumptions on the weight function A in Moynot and Samuelides allow the 
Radon-Nikodym derivative to be tensored. A very important implication of 
our result is that the mean-field behaviour is insufficient to characterise the 
behaviour of a population. Our limit process \i e is system-wide and ergodic. 
Our work challenges the assumption held by some that one cannot have a 
'concise' macroscopic description of a neural network without an assumption 
of asynchronicity at the local population level. 

The utility of this paper extends well beyond the identification of the limit 
law fi e . The LDP provides a powerful means of assessing how quickly the 
empirical measure converges to its limit. In particular, it provides a means 
of assessing the probability of finite size effects. For example if it could 
be shown that the rate function H is sharply convex everywhere, then one 
would be more confident that the system converges quickly to its limit law. 
The rate functions of many classical LDPs, such as the one in lemma EH are 

6 We noted in the introduction that this is termed propagation of chaos by some. 
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indeed convex (in fact the rate function H u (-), for fixed v, is affine). However 
it is not clear whether our rate function H is convex. Indeed if it could be 
shown that the rate function H is not sharply convex, and in particular that 
it has a local minimum at another point /i m , then perhaps if N is not too 
great there could be a reasonable probability that the empirical measure lies 
close to fi m . The upshot of this discussion is that further exploration of the 
topology of the rate function H could be a very fruitful avenue of research 
for assessing the probability of finite-size effects. It would be of interest to 
compare our LDP with other analyses of the rate of convergence of neural 
networks to their limits as the size asymptotes to infinity. This includes the 
system-size expansion of Bressloff [3] , the path-integral formulation of Buice 
and Cowan [B] and the systematic expansion of the moments by (amongst 
others) [281 E21 E] - 
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A A comment on lemma 1321 

We firstly suppose that c u = 0, so that Q u is a centred stationary Gaussian 
Process. We denote the corresponding image law by . There exists an 
LDP for n 1 ^ with good rate function 

Ste) = I (3 \bE?) ~ 2^(ll«°ll 2 ) - (T + 1) loga+ 

— ^ f (Id T+1 - A"(u)) : v^duj) + -L f log det(a 2 Id T+1 + K v {u)) du. 

(94) 

If /i ^ £2, then the spectral density does not exist (as we noted in (1301) ) 
and Hq is infinite. We now comment on how this expression (I94|) is obtained. 
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The existence of an LDP for n^ ,7V was proved by Baxter and Jain [2] , al- 
though they did not provide an explicit expression for the rate function. The 
above expression for the rate function may be obtained through a straight- 
forward extension of the proof in Donsker and Varadhan [21]. Donsker and 
Varadhan proved their expression (labelled (1.9) in their paper) in the case 
of R a , finding (in their notation) 



H f (R) = E 



R 



r(y\u) \ogr(y\u)dy 

i 

»2tt 



+ 



Air 



- log 2?r 
dG{i) 1 



2 k 



log/( 7 )d 7 . (95) 



Here R is a stationary measure in .M^IR 2 ) with regular conditional prob- 
ability distribution r(y\u) and continuous spectral density G{^f). The sta- 
tionary Gaussian process against which the entropy is taken has continuous 
spectral density f(j) : [0, 2tt] — > R and zero mean. We briefly explain how 
this expression corresponds to ours. 

Donsker and Varadhan obtained their expression Hf(R) through a similar 
technique to ours: they 'diagonalise' the covariance operator using a 'moving 
average' transformation (note that the diagonalised variables in Donsker and 
Varadhan have variance 1, whereas in our case they have variance a 2 ). It is 
easily shown that the transformed operator (which is analogous to our P z ) 
satisfies an LDP because the variables are independent and identically dis- 
tributed. Recall that, in our model, the entropy of the 'diagonalised' system 
is J( 3 )(/i,P z ) = limjv^oo A r_1 J (2) (/i Ar , p®^). In fact the terms I® - 

7r^-E=[||i; || 2 ] in (J94l) correspond to the terms ~E R { f ™ r(y\uj) \ogr(y\uj)dy] + 



2(7 



|log27r in (195]) (this may be inferred from the expression (199]) for I^(jj N : P® N )). 
The expression 



47TCT 2 



(Id r+ i - A"(u)) : v»{du) 



lim N- 1 ^ 



7V->oo 



(Id 



N(T+1) 



A V,N ) 



may be thought of as the asymptotic limit of the expectation of the quadratic 
form induced by the inverse covariance operator. It corresponds to j- J Q 27r ' 
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in (I95p . Finally, the terms in ( I94p of the form 

J log det(a 2 Id (T+ i) + K v {u)) dw - (T + 1) log(cr) 

- hm V 1 lor ( d6t (^^D + RV ' N ) \ - y (V ) 

are the asymptotic limit of the logarithm of the ratio of the determinant of 
the original system divided by the determinant of the 'diagonalised' system. 
They corresponds to the term j- J Q 27r log 7(7)^7 in (195|) . The extension of the 
proof in Donsker and Varadhan to our case is straightforward because S = 
R T+1 is finite-dimensional and iV^logdet (<r 2 Idjv(T+i) + K U ' N } is bounded 
below by 2 log a, although we must omit the details because of a shortage of 
space. 

We now use the rate function fl94"j) governing the zero-mean process to 
establish an LDP for a process with nonzero mean c u . Let O : M± S (S' Z ) — > 
M^ S (S 7L ) be the translation map, such that for measureable A, Q(fi)(A) = 
li(A + c u ). Since 

W [v ot v k ] = E @M [ (v° - c v ) 1 (v k -c% (96) 

we find that 

v"{du) = v @M (du) + 2tt5(uj) (c u V - c u E @M [V] - E 0( ^ [v°] 'c") . (97) 

Since ft N (v~ n , • ■ ■ , v n ) is the image of fi N (v ~ n — c u , ■ ■ • , v n — c u ) under G, 
it follows from the Contraction Principle that there exists an LDP for IV /,N 
with rate function H_"(fi) := i^j(@ _1 (/i))- Since A v (0) is symmetric, we may 
infer from (EH) and (l97j that 



ST (ft) = /^(e- 1 ^),^) - f I» : v»{du) 

l — { t c v A v {^)c v - 2 t c u A u (0)¥^[v ])+— [ logdet(ld T +i+a- 2 k u (cu))duj. 
2a 2 \ /An 



(98) 



We now determine an explicit expression for /( 3 )(0 1 (yu),P z ). If fi N does 
not possess a density then I^(/jl n , p® N } is infinite (because P® N possesses a 
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density, and therefore fx N is not absolutely continuous with respect to P 0JV ) 
Otherwise, let the density of fj, N be \v~ n , . . . , v n ). We find that 

lM(ff,gn= I log [ - {V) l ) r^(v)dv, (99) 



(2trx 2 ) 2 exp(-^2 



where dv = YYj=- n Yl^=o ^ v i- Upon expansion, we find that 
I^(Q- 1 (jf),g 9N ) = j (log (r^(v + c v ] 



■^2^1og( 27 ra 2 ) + ^\\v\A r» N (v + <?)dv. 



Thus 



l( 2 \Q-\fi N ),P® N ) = I^(fi N ,P® N ) - ^(c",m[v }) - ^V\\ 2 . (100) 
= — = — a 1 2o~ 2 

Noting that ti°] = (definition Hj), the identity (J77j) now follows from 
([63]), (J72D, ([73]), (gM) and (HDD]). 



B Proof of Proposition 

Proof. We have already proved that I\ is continuous in proposition [T9j It 
thus suffices for us to prove that for some constant Cj, 

\r»M - tm\ < Cje (i + /( 3 )(/i, p s )) . (ioi) 

Fix e. We define <$> N (x) to be the integrand of CN(T 2 (/i N ) - T v 2 {n N )) for 
some positive constant C . Taking the expression from the proof of lemma 
we have 



\ l=-n 



Here y is defined in (1811) . This is a continuous real function on T N which is 
unbounded. We have that 

*" S ( T?F Yl W"' - Z"*\\\Wt + 2(d„,„,y") + N\e„J 



£ II* 

\ Z=-ra 
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In turn, using the identities in lemma I3"51 we find that $ < $„ a:l . where 
C 



max 2(T^ 



Note that $^ aa , is integrable with respect to \i because \i G For M > 0, 
let 

Bn\m = {V ■ <aM < or |M| 2 < iVAf}. 
It may be observed that B^ M is compact, because the eigenvalues of the 
quadratic form in $^ ai are strictly positive. It follows that $ N and $U ai are 
bounded over B u / M (for all M). Let $^ be ^ multiplied by the indicator 
function over the set B V jf M . Since $^ is bounded and continuous, we find 
from the Fenchel-Legendre transform that 

*S(y) n N {dx) < log / exp^^P^^ + J^^P^). (102) 
In order that J TN exp^^ iax (y)P' g>N (dx) < oo, we stipulate that 

C*(C^ + £ )' 

for some constant < C\ < 1, where C* is given in lemma |35j Since 
< $max over -^jvm 5 we ma y a PPly the dominated convergence theorem 
to ffT02|) (taking M ^ oo) to obtain 

<t> N (y)» N (dx) < log f e W ^ ax (y)P® N (dx) + lW(v N ,P® N )- 

We observe from proposition [1] and equation (182]) that, under the trans- 
formation h : w — >■ P® N (dv) becomes 

o h-\dy) = (g) jV(0r+i, JV<7 2 Id r+ i)<V. 

l=—n 

An application of lemma [6] thus yields 



/ exp^ ax (y)P^(dx)=exp 
Jt n 



x 



2a 2 

N(T+1) NC 2 



;i - r,j .- x ex P 2a ;-_ \\d u j 2 . (103) 
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We use the fact (proved in lemma I3"5T) that \\d u ^\\ < C*(C^ +e) to find that 
NC(T 2 (p N ) - T u 2 (fi N )) <Ns + J (2) (/A P® N ) 

where 

s = 7T~ ^l 1 - C lj 



2a 2 2 toV " 2cr 2 (l-C 1 )' 

We divide by NC and take the limit as N — > oo. The result f llOip follows 
since, by lemma EU C V N — > as — oo. □ 



References 

[1] J. Baladron, D. Fasoli, and O. Faugeras, Three applications of 
gpu computing in neuroscience, Computing in Science and Engineering, 
(2012). 

[2] J. R. Baxter and N. C. Jain, An approximation condition for large 
deviations and some applications, in Convergence in Ergodic Theory 
and Probability, V. Bergulson, ed., Ohio State University Mathematical 
Research Institute Publications, 1993. 

[3] G. Ben-Arous AND A. Guionnet, Large deviations for langevin spin 
glass dynamics, Probability Theory and Related Fields, 102 (1995), 
pp. 455-509. 

[4] , Symmetric Langevin Spin Glass Dynamics, The Annals of Proba- 
bility, 25 (1997), pp. 1367-1422. 

[5] P. Bressloff, Stochastic neural field theory and the system-size ex- 
pansion, SIAM J. Appl. Math, 70 (2009), pp. 1488-1521. 

[6] M. Buice and J. Cowan, Field-theoretic approach to fluctuation ef- 
fects in neural networks, Physical Review E, 75 (2007). 

[7] M. Buice, J. Cowan, and C. Chow, Systematic fluctuation ex- 
pansion for neural network activity equations, Neural computation, 22 
(2010), pp. 377-426. 

[8] M. J. Caceres, J. A. Carillo, and B. Perhame, Analysis of 
nonlinear noisy integrate and fire neuron models: blow-up and steady 
states, The Journal of Mathematical Neuroscience, 1 (2011). 



70 



B. CESSAC, Increase in complexity in random neural networks, Journal 
de Physique I (France), 5 (1995), pp. 409-432. 

B. CESSAC AND M. SAMUELIDES, From neuron to neural networks 
dynamics., EPJ Special topics: Topics in Dynamical Neural Networks, 
142 (2007), pp. 7-88. 

A. Crisanti and H. Sompolinsky, Dynamics of spin systems with 
randomly asymmetric bonds: Langevin dynamics and a spherical model, 
Physical Review A, 36 (1987), pp. 4922-4939. 

, Dynamics of spin systems with randomly asymmetric bounds: Ising 



spins and Glauber dynamics, Phys. Review A, 37 (1987), p. 4865. 

L. F. CUGLIANDOLO, J. KURCHAN, P. Le DOUSSAL, AND L. PELITI, 

Glassy behaviour in disordered systems with nonrelaxational dynamics, 
Physical review letters, 78 (1997), pp. 350-353. 

D. Daley and D. Vere-Jones, An introduction to the theory of point 
processes: volume II: general theory and structure, vol. 2, Springer, 2007. 

D. Dawson and P. Del Moral, Large deviations for interacting 
processes in the strong topology, in Statistical Modeling and Analysis for 
Complex Data Problems, Springer US, 2005. 

D. Dawson and J. Gartner, Large deviations from the mckean- 
vlasov limit for weakly interacting diffusions, Stochastics, 20 (1987). 

, Multilevel large deviations and interacting diffusions, Probability 

Theory and Related Fields, 98 (1994), pp. 423-487. 

P. Dayan and L. Abbott, Theoretical N euro science : Computational 
and Mathematical Modeling of Neural Systems, MIT Press, 2001. 

A. Dembo and O. Zeitouni, Large deviations techniques, Springer, 
1997. 2nd Edition. 

J.-D. Deuschel and D. W. Stroock, Large Deviations, vol. 137 of 
Pure and Applied Mathematics, Academic Press, 1989. 

M. Donsker AND S. Varadhan, Large deviations for stationary 
Gaussian processes, Commun. Math. Phys., 97 (1985), pp. 187-210. 



71 



S. ElBoustani and A. Destexhe, A master equation formalism for 
macroscopic modeling of asynchronous irregular activity states, Neural 
computation, 21 (2009), pp. 46-100. 

R. Ellis, Entropy, large deviations and statistical mechanics, Springer, 
1985. 

G. B. Ermentrout and D. Terman, Foundations of Mathematical 
Neuroscience, Interdisciplinary Applied Mathematics, Springer, 2010. 

W. Gerstner, Time structure of the activity in neural network models, 
Physical Review E, 51 (1995), pp. 738-758. 

W. Gerstner and W. Kistler, Spiking Neuron Models, Cambridge 
University Press, 2002. 

W. Gerstner and J. van Hemmen, Coherence and incoherence in a 
globally coupled ensemble of pulse- emitting units, Physical review letters, 
71 (1993), pp. 312-315. 

I. GlNZBURG AND H. SOMPOLINSKY, Theory of correlations in stochas- 
tic neural networks, Physical Review E, 50 (1994). 

A. Guionnet, Dynamique de Langevin d'un verre de spins, PhD thesis, 
Universite de Paris Sud, 1995. 

, Averaged and quenched propagation of chaos for spin glass dynam- 



ics, Probability Theory and Related Fields, 109 (1997), pp. 183-215. 

E. Izhikevich, Dynamical Systems in Neuroscience: The Geometry of 
Excitability And Bursting, MIT Press, 2007. 

E. Kandel, J. Schwartz, and T. Jessel, Principles of Neural Sci- 
ence, McGraw-Hill, 4th ed., 2000. 

L. Lapicque, Recherches quantitatifs sur V excitation des nerfs traitee 
comme une polarisation, J. Physiol. Paris, 9 (1907), pp. 620-635. 

O. Moynot, Etude mathematique de la dynamique des reseaux neu- 
ronaux aleatoires recurrents, PhD thesis, Universite Paul Sabatier, 
Toulouse, 1999. 



72 



[35] O. MOYNOT AND M. SAMUELIDES, Large deviations and mean-field 
theory for asymmetric random recurrent neural networks, Probability 
Theory and Related Fields, 123 (2002), pp. 41-75. 

[36] J. Neveu, Processus aleatoires gaussiens, vol. 34, Presses de l'Universite 
de Montreal, 1968. 

[37] M. Samuelides and B. Cessac, Random recurrent neural networks, 
European Physical Journal - Special Topics, 142 (2007), pp. 7-88. 

[38] H. Sompolinsky, A. Crisanti, and H. Sommers, Chaos in Random 
Neural Networks, Physical Review Letters, 61 (1988), pp. 259-262. 

[39] H. Sompolinsky and A. Zippelius, Dynamic theory of the spin-glass 
phase, Physical Review Letters, 47 (1981), pp. 359-362. 

[40] , Relaxational dynamics of the Edwards-Anderson model and 

the mean-field theory of spin-glasses, Physical Review B, 25 (1982), 
pp. 6860-6875. 



73 



